DOMAIN: Electronics and Telecommunication
CONTEXT: A communications equipment manufacturing company has a product which is responsible for emitting informative signals. Company wants to build a machine learning model which can help the company to predict the equipment’s signal quality using various parameters.
DATA DESCRIPTION: The data set contains information on various signal tests performed:
1. Parameters: Various measurable signal parameters.
2. Signal_Quality: Final signal strength or quality
PROJECT OBJECTIVE: The need is to build a regressor which can use these parameters to determine the signal strength or quality [as number]. Steps and tasks:
1. Import data.
2. Data analysis & visualisation
• Perform relevant and detailed statistical analysis on the data.
• Perform relevant and detailed uni, bi and multi variate analysis.
Hint: Use your best analytical approach. Even you can mix match columns to create new columns which can be used for better analysis. Create your own features if required. Be highly experimental and analytical here to find relevant hidden patterns.
3. Design, train, tune and test a neural network regressor.
Hint: Use best approach to refine and tune the data or the model. Be highly experimental here.
4. Pickle the model for future use
#Importing required libraries
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dense
from tensorflow.keras.layers import BatchNormalization, Dropout
from tensorflow.keras import optimizers
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import sklearn
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
etRawFile = pd.read_csv("./Part- 1 - Signal.csv")
• Perform relevant and detailed statistical analysis on the data.
• Perform relevant and detailed uni, bi and multi variate analysis.
Reading sample data
etRawFile.sample(3)
| Parameter 1 | Parameter 2 | Parameter 3 | Parameter 4 | Parameter 5 | Parameter 6 | Parameter 7 | Parameter 8 | Parameter 9 | Parameter 10 | Parameter 11 | Signal_Strength | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1529 | 7.1 | 0.69 | 0.08 | 2.1 | 0.063 | 42.0 | 52.0 | 0.99608 | 3.42 | 0.60 | 10.2 | 6 |
| 265 | 11.8 | 0.26 | 0.52 | 1.8 | 0.071 | 6.0 | 10.0 | 0.99680 | 3.20 | 0.72 | 10.2 | 7 |
| 1570 | 6.4 | 0.36 | 0.53 | 2.2 | 0.230 | 19.0 | 35.0 | 0.99340 | 3.37 | 0.93 | 12.4 | 6 |
Checking data types and other information
#check the data types, size of dataframe in mem
etRawFile.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1599 entries, 0 to 1598 Data columns (total 12 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Parameter 1 1599 non-null float64 1 Parameter 2 1599 non-null float64 2 Parameter 3 1599 non-null float64 3 Parameter 4 1599 non-null float64 4 Parameter 5 1599 non-null float64 5 Parameter 6 1599 non-null float64 6 Parameter 7 1599 non-null float64 7 Parameter 8 1599 non-null float64 8 Parameter 9 1599 non-null float64 9 Parameter 10 1599 non-null float64 10 Parameter 11 1599 non-null float64 11 Signal_Strength 1599 non-null int64 dtypes: float64(11), int64(1) memory usage: 150.0 KB
View some basic statistical details like percentile, mean, std etc.
etRawFile.describe().T
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| Parameter 1 | 1599.0 | 8.319637 | 1.741096 | 4.60000 | 7.1000 | 7.90000 | 9.200000 | 15.90000 |
| Parameter 2 | 1599.0 | 0.527821 | 0.179060 | 0.12000 | 0.3900 | 0.52000 | 0.640000 | 1.58000 |
| Parameter 3 | 1599.0 | 0.270976 | 0.194801 | 0.00000 | 0.0900 | 0.26000 | 0.420000 | 1.00000 |
| Parameter 4 | 1599.0 | 2.538806 | 1.409928 | 0.90000 | 1.9000 | 2.20000 | 2.600000 | 15.50000 |
| Parameter 5 | 1599.0 | 0.087467 | 0.047065 | 0.01200 | 0.0700 | 0.07900 | 0.090000 | 0.61100 |
| Parameter 6 | 1599.0 | 15.874922 | 10.460157 | 1.00000 | 7.0000 | 14.00000 | 21.000000 | 72.00000 |
| Parameter 7 | 1599.0 | 46.467792 | 32.895324 | 6.00000 | 22.0000 | 38.00000 | 62.000000 | 289.00000 |
| Parameter 8 | 1599.0 | 0.996747 | 0.001887 | 0.99007 | 0.9956 | 0.99675 | 0.997835 | 1.00369 |
| Parameter 9 | 1599.0 | 3.311113 | 0.154386 | 2.74000 | 3.2100 | 3.31000 | 3.400000 | 4.01000 |
| Parameter 10 | 1599.0 | 0.658149 | 0.169507 | 0.33000 | 0.5500 | 0.62000 | 0.730000 | 2.00000 |
| Parameter 11 | 1599.0 | 10.422983 | 1.065668 | 8.40000 | 9.5000 | 10.20000 | 11.100000 | 14.90000 |
| Signal_Strength | 1599.0 | 5.636023 | 0.807569 | 3.00000 | 5.0000 | 6.00000 | 6.000000 | 8.00000 |
Parameter 1:
mean and median value are close by, so the data seems to be normally distributed. There is considerable difference between 75% and max, so there could be outiers. Need to check modality of the data.
Parameter 2:
mean and median values are close by, so the data seems to be normally distributed. There could be few outiers, we need to check data using boxplot.
Parameter 3:
mean and median values are close by, so the data seems to be normally distributed. There could be few outiers, we need to check data using boxplot.
Parameter 4:
mean and median values are close by, so the data seems to be normally distributed. There is considerable difference between 75% and max, so there could be outiers.
Parameter 5:
mean and median values are close by, so the data seems to be normally distributed. There is considerable difference between 75% and max, so there could be outiers.
Parameter 6:
There is considerable difference between 75% and max, so there could be outiers. Need to check the distribution of data in kde plot.
Parameter 7:
There is considerable difference between 75% and max, so there could be outiers. Need to check the distribution of data in kde plot.
Parameter 8:
mean and median value are close by, so the data seems to be normally distributed. There is considerable difference between 75% and max, so there could be outiers.
Parameter 9:
mean and median value are close by, so the data seems to be normally distributed. There is considerable difference between 75% and max, so there could be outiers.
Parameter 10:
mean and median value are close by, so the data seems to be normally distributed. There is considerable difference between 75% and max, so there could be outiers.
meanMedianComparison = etRawFile.describe().T[['mean','50%']]
meanMedianComparison['mean']-meanMedianComparison['50%']
Parameter 1 0.419637 Parameter 2 0.007821 Parameter 3 0.010976 Parameter 4 0.338806 Parameter 5 0.008467 Parameter 6 1.874922 Parameter 7 8.467792 Parameter 8 -0.000003 Parameter 9 0.001113 Parameter 10 0.038149 Parameter 11 0.222983 Signal_Strength -0.363977 dtype: float64
Above table would give insight into data about distribution of mean and median for each column.
#check if the values are null
etRawFile.isnull().sum()
Parameter 1 0 Parameter 2 0 Parameter 3 0 Parameter 4 0 Parameter 5 0 Parameter 6 0 Parameter 7 0 Parameter 8 0 Parameter 9 0 Parameter 10 0 Parameter 11 0 Signal_Strength 0 dtype: int64
There are no null values observed in the above cell output, so no need to null value correction.
Univariate data analysis is done below.
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
fig, ax = plt.subplots(5,2, figsize=(18,18))
listOfColumns = etRawFile.columns[:10]
len_listOfColumns = len(listOfColumns)
index = 0
for j in range(len(ax)):
for i in range(len(ax[j])):
if(index < len_listOfColumns):
ax[j][i] = sns.distplot(etRawFile[listOfColumns[index]],bins=20,rug=True,ax = ax[j][i]);
index = index + 1
fig.show()
D:\anaconda\envs\gl\lib\site-packages\ipykernel_launcher.py:14: UserWarning: Matplotlib is currently using module://ipykernel.pylab.backend_inline, which is a non-GUI backend, so cannot show the figure.
As seen in above diagram, most columns are having normal distribution of data. There are modality obsereved in parameter 3 and parameter 2 column.
Long tail observed for Parameter 4, Parameter 5,Parameter 6, Parameter 7 and Parameter 10 and can be considered as right skewed data distribution. For these column we need to check outliers and treat them as required.
Viewing data in boxplot would help to identify the outliers.
etRawFile.boxplot(figsize=(35,15));
Since there are outliers observed on many column, let's see the data in detail box plot for individual columns.
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
fig, ax = plt.subplots(5,2, figsize=(18,18))
listOfColumns = etRawFile.columns[:10]
len_listOfColumns = len(listOfColumns)
index = 0
for j in range(len(ax)):
for i in range(len(ax[j])):
if(index < len_listOfColumns):
ax[j][i] = sns.boxplot(y=etRawFile[listOfColumns[index]],showmeans=True, ax = ax[j][i]);
ax[j][i].axhline(etRawFile[listOfColumns[index]].mean(),color = 'r', ls='--')
index = index + 1
fig.show()
D:\anaconda\envs\gl\lib\site-packages\ipykernel_launcher.py:15: UserWarning: Matplotlib is currently using module://ipykernel.pylab.backend_inline, which is a non-GUI backend, so cannot show the figure. from ipykernel import kernelapp as app
There are outliers observed in every column which are outiers of 1.5*IQR. We shall run our final model on both outlier treated data and the one without treating it and observe the output.
Azme Khamis, Zuhaimy Ismail , Khalid Haron and Ahmad Tarmizi Mohammed , 2005. The Effects of Outliers Data on Neural Network Performance. Journal of Applied Sciences, 5: 1394-1398.
DOI: 10.3923/jas.2005.1394.1398
dfO = etRawFile.iloc[:,:-1]
Q1 = dfO.quantile(0.25)
Q3 = dfO.quantile(0.75)
IQR = Q3 - Q1
perOfOutliers = ((((dfO < (Q1 - 1.5 * IQR)) | (dfO > (Q3 + 1.5 * IQR))).sum())/dfO.shape[0]*100).to_frame()
perOfOutliers
| 0 | |
|---|---|
| Parameter 1 | 3.064415 |
| Parameter 2 | 1.188243 |
| Parameter 3 | 0.062539 |
| Parameter 4 | 9.693558 |
| Parameter 5 | 7.004378 |
| Parameter 6 | 1.876173 |
| Parameter 7 | 3.439650 |
| Parameter 8 | 2.814259 |
| Parameter 9 | 2.188868 |
| Parameter 10 | 3.689806 |
| Parameter 11 | 0.813008 |
Percentage of outliers are very less, so we observe that these may not affect the output of Neural network
print("Count of outliers ::\n", ((dfO < (Q1 - 1.5 * IQR)) | (dfO > (Q3 + 1.5 * IQR))).sum())
Count of outliers :: Parameter 1 49 Parameter 2 19 Parameter 3 1 Parameter 4 155 Parameter 5 112 Parameter 6 30 Parameter 7 55 Parameter 8 45 Parameter 9 35 Parameter 10 59 Parameter 11 13 dtype: int64
#Correlation Matrix
df = etRawFile
corr = df.corr() # correlation matrix
lower_triangle = np.tril(corr, k = -1)
mask = lower_triangle == 0 # to mask the upper triangle in the following heatmap
plt.figure(figsize = (15,8))
sns.set_style(style = 'white') # Setting it to white so that we do not see the grid lines
sns.heatmap(lower_triangle, center=0.5, cmap= 'Blues', annot= True, xticklabels = corr.index, yticklabels = corr.columns, cbar= False, linewidths= 1, mask = mask)
plt.xticks(rotation = 30)
plt.yticks(rotation = 20)
plt.show()
As observed the correlation between any 2 variable is not strong (here more than 85 % can be considered as strong), so there is no need to delete any columns.
g = sns.pairplot(etRawFile, diag_kind='kde')
g = sns.pairplot(etRawFile
,diag_kind='kde'
,hue = 'Signal_Strength'
,palette = 'rocket' # #69d
,vars=['Parameter 1','Parameter 3','Parameter 6','Parameter 7','Parameter 8','Parameter 9','Parameter 10']
)
g.map_lower(sns.kdeplot, levels=4, color=".2")
D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws, D:\anaconda\envs\gl\lib\site-packages\seaborn\distributions.py:1185: UserWarning: No contour levels were found within the data range. **contour_kws,
<seaborn.axisgrid.PairGrid at 0x24db1c1ff98>
Observing above graphs, we can see how the density distribution of the data. Paremeter 4,5,6 have denser concentration of data. These columns could be drivers of the regression. As we see in kde, the data points seems to give good variety of the data.
import warnings
warnings.simplefilter("ignore", UserWarning)
# etRawFile.columns
for i in ['Parameter 1', 'Parameter 2', 'Parameter 3', 'Parameter 4',
'Parameter 5', 'Parameter 6', 'Parameter 7', 'Parameter 8',
'Parameter 9', 'Parameter 10', 'Parameter 11']:
g = sns.catplot(#x="Signal_Strength",
y = i ,
col="Signal_Strength",
data=etRawFile, height=5, aspect=2,
s=20,alpha=0.5, linewidth=3,
kind="swarm"
);
g.set_xticklabels(rotation=30);
In above graphs, we can observe that the data distribution is having wide range for signal strength - 5,6,7.
For other 3 types of signal strength, the data distribution is slim.
Although this is a regression problem and Signal strength is considered as a continous variable, there are only few values for signal strength. These signal strength if we consider them as a categorical column for time being, the above graph gives some insight into data.
Making copy of the raw file, which will be used for netwok training.
etRawFileOutliersRemoved = etRawFile.copy()
#find the outliers and replace them by median
for col_name in etRawFile.columns[:-1]:
q1 = etRawFile[col_name].quantile(0.25)
q3 = etRawFile[col_name].quantile(0.75)
iqr = q3 - q1
low = q1-1.5*iqr
high = q3+1.5*iqr
etRawFileOutliersRemoved.loc[(etRawFileOutliersRemoved[col_name] < low) | (etRawFileOutliersRemoved[col_name] > high), col_name] = etRawFileOutliersRemoved[col_name].median()
etRawFileOutliersRemoved.boxplot(figsize=(35,15));
etRawFile.boxplot(figsize=(35,15));
As we can observe, the outliers are reduced in the etRawFileOutliersRemoved dataset. We shall use this in the network to see if outlier are affecting the performance.
Preparing dataset for training the network
#getting the features of the dataset
etFeatures=etRawFile.iloc[:,:-1].values
#getting the target value of the dataset
etActualSignal = etRawFile.iloc[:,-1].values
print('Number of examples: ', etFeatures.shape[0])
print('Number of features for each example: ', etFeatures.shape[1])
print('Shape of actual prices data: ', etActualSignal.shape)
Number of examples: 1599 Number of features for each example: 11 Shape of actual prices data: (1599,)
X_train_Features, X_test_Features, y_train_ActualSignal, y_test_ActualSignal = train_test_split(etFeatures,etActualSignal, test_size=0.2, random_state=42)
print('Number of Train examples: ', X_train_Features.shape[0])
print('Number of Train features for each example: ', X_train_Features.shape[1])
print('Shape of Train actual prices data: ', y_train_ActualSignal.shape)
print('Number of Test examples: ', X_test_Features.shape[0])
print('Number of Test features for each example: ', X_test_Features.shape[1])
print('Shape of Test actual prices data: ', y_test_ActualSignal.shape)
Number of Train examples: 1279 Number of Train features for each example: 11 Shape of Train actual prices data: (1279,) Number of Test examples: 320 Number of Test features for each example: 11 Shape of Test actual prices data: (320,)
X_train_Features[:5]
array([[8.7000e+00, 6.9000e-01, 3.1000e-01, 3.0000e+00, 8.6000e-02,
2.3000e+01, 8.1000e+01, 1.0002e+00, 3.4800e+00, 7.4000e-01,
1.1600e+01],
[6.1000e+00, 2.1000e-01, 4.0000e-01, 1.4000e+00, 6.6000e-02,
4.0500e+01, 1.6500e+02, 9.9120e-01, 3.2500e+00, 5.9000e-01,
1.1900e+01],
[1.0900e+01, 3.9000e-01, 4.7000e-01, 1.8000e+00, 1.1800e-01,
6.0000e+00, 1.4000e+01, 9.9820e-01, 3.3000e+00, 7.5000e-01,
9.8000e+00],
[8.8000e+00, 6.8500e-01, 2.6000e-01, 1.6000e+00, 8.8000e-02,
1.6000e+01, 2.3000e+01, 9.9694e-01, 3.3200e+00, 4.7000e-01,
9.4000e+00],
[8.4000e+00, 1.0350e+00, 1.5000e-01, 6.0000e+00, 7.3000e-02,
1.1000e+01, 5.4000e+01, 9.9900e-01, 3.3700e+00, 4.9000e-01,
9.9000e+00]])
y_train_ActualSignal[:5]
array([6, 6, 6, 5, 5], dtype=int64)
#getting the features of the dataset
etRawFileOutliersRemovedFeatures = etRawFileOutliersRemoved.iloc[:,:-1].values
#getting the target value of the dataset
etRawFileOutliersRemovedActualSignal = etRawFileOutliersRemoved.iloc[:,-1].values
print('Number of examples: ', etRawFileOutliersRemovedFeatures.shape[0])
print('Number of features for each example: ', etRawFileOutliersRemovedFeatures.shape[1])
print('Shape of actual prices data: ', etRawFileOutliersRemovedActualSignal.shape)
Number of examples: 1599 Number of features for each example: 11 Shape of actual prices data: (1599,)
X_train_OutliersRemovedFeatures, X_test_OutliersRemovedFeatures, y_train_OutliersRemovedActualSignal, y_test_OutliersRemovedActualSignal = train_test_split(etRawFileOutliersRemovedFeatures,etRawFileOutliersRemovedActualSignal, test_size=0.2, random_state=42)
print('Number of Train examples: ', X_train_OutliersRemovedFeatures.shape[0])
print('Number of Train features for each example: ', X_train_OutliersRemovedFeatures.shape[1])
print('Shape of Train actual prices data: ', y_train_OutliersRemovedActualSignal.shape)
print('Number of Test examples: ', X_test_OutliersRemovedFeatures.shape[0])
print('Number of Test features for each example: ', X_test_OutliersRemovedFeatures.shape[1])
print('Shape of Test actual prices data: ', y_test_OutliersRemovedActualSignal.shape)
Number of Train examples: 1279 Number of Train features for each example: 11 Shape of Train actual prices data: (1279,) Number of Test examples: 320 Number of Test features for each example: 11 Shape of Test actual prices data: (320,)
X_train_OutliersRemovedFeatures[:5]
array([[8.7000e+00, 6.9000e-01, 3.1000e-01, 3.0000e+00, 8.6000e-02,
2.3000e+01, 8.1000e+01, 1.0002e+00, 3.4800e+00, 7.4000e-01,
1.1600e+01],
[6.1000e+00, 2.1000e-01, 4.0000e-01, 1.4000e+00, 6.6000e-02,
4.0500e+01, 3.8000e+01, 9.9675e-01, 3.2500e+00, 5.9000e-01,
1.1900e+01],
[1.0900e+01, 3.9000e-01, 4.7000e-01, 1.8000e+00, 1.1800e-01,
6.0000e+00, 1.4000e+01, 9.9820e-01, 3.3000e+00, 7.5000e-01,
9.8000e+00],
[8.8000e+00, 6.8500e-01, 2.6000e-01, 1.6000e+00, 8.8000e-02,
1.6000e+01, 2.3000e+01, 9.9694e-01, 3.3200e+00, 4.7000e-01,
9.4000e+00],
[8.4000e+00, 5.2000e-01, 1.5000e-01, 2.2000e+00, 7.3000e-02,
1.1000e+01, 5.4000e+01, 9.9900e-01, 3.3700e+00, 4.9000e-01,
9.9000e+00]])
y_train_OutliersRemovedActualSignal[:5]
array([6, 6, 6, 5, 5], dtype=int64)
Training a simple network to see if we are able to find the proper regression, we shall observe the results and decide further development of the network.
def simple_ET_model():
# Initialize Sequential model
model = tf.keras.models.Sequential()
# Normalize input data
model.add(tf.keras.layers.BatchNormalization(input_shape=(11,)))
# Add final Dense layer for prediction - Tensorflow.keras declares weights and bias automatically
model.add(tf.keras.layers.Dense(1, kernel_initializer='normal',activation='relu'))
# Compile the model - add mean squared error as loss and stochastic gradient descent as optimizer
model.compile(optimizer='adam', loss='mse')
return model
simple_model = simple_ET_model()
simple_model.fit(X_train_Features, y_train_ActualSignal, epochs=150, validation_split=0.35)
Epoch 1/150 26/26 [==============================] - 2s 14ms/step - loss: 31.2001 - val_loss: 23.0700 Epoch 2/150 26/26 [==============================] - 0s 3ms/step - loss: 30.4982 - val_loss: 22.4391 Epoch 3/150 26/26 [==============================] - 0s 3ms/step - loss: 29.6616 - val_loss: 21.9168 Epoch 4/150 26/26 [==============================] - 0s 4ms/step - loss: 28.6117 - val_loss: 21.2689 Epoch 5/150 26/26 [==============================] - 0s 3ms/step - loss: 27.4081 - val_loss: 20.3311 Epoch 6/150 26/26 [==============================] - 0s 3ms/step - loss: 26.0620 - val_loss: 19.3179 Epoch 7/150 26/26 [==============================] - 0s 5ms/step - loss: 24.4931 - val_loss: 18.1007 Epoch 8/150 26/26 [==============================] - 0s 3ms/step - loss: 22.8482 - val_loss: 16.7526 Epoch 9/150 26/26 [==============================] - 0s 4ms/step - loss: 21.0677 - val_loss: 15.3852 Epoch 10/150 26/26 [==============================] - 0s 3ms/step - loss: 19.3125 - val_loss: 13.9925 Epoch 11/150 26/26 [==============================] - 0s 3ms/step - loss: 17.5559 - val_loss: 12.6323 Epoch 12/150 26/26 [==============================] - 0s 4ms/step - loss: 15.7735 - val_loss: 11.2710 Epoch 13/150 26/26 [==============================] - 0s 5ms/step - loss: 14.1756 - val_loss: 10.0030 Epoch 14/150 26/26 [==============================] - 0s 3ms/step - loss: 12.5830 - val_loss: 8.7965 Epoch 15/150 26/26 [==============================] - 0s 3ms/step - loss: 10.9669 - val_loss: 7.6606 Epoch 16/150 26/26 [==============================] - 0s 3ms/step - loss: 9.6177 - val_loss: 6.5882 Epoch 17/150 26/26 [==============================] - 0s 3ms/step - loss: 8.3118 - val_loss: 5.7081 Epoch 18/150 26/26 [==============================] - 0s 3ms/step - loss: 7.2812 - val_loss: 4.9418 Epoch 19/150 26/26 [==============================] - 0s 2ms/step - loss: 6.2857 - val_loss: 4.3131 Epoch 20/150 26/26 [==============================] - 0s 3ms/step - loss: 5.4628 - val_loss: 3.7951 Epoch 21/150 26/26 [==============================] - 0s 3ms/step - loss: 4.8089 - val_loss: 3.4080 Epoch 22/150 26/26 [==============================] - 0s 3ms/step - loss: 4.3285 - val_loss: 3.0641 Epoch 23/150 26/26 [==============================] - 0s 3ms/step - loss: 3.9159 - val_loss: 2.8016 Epoch 24/150 26/26 [==============================] - 0s 3ms/step - loss: 3.5352 - val_loss: 2.5701 Epoch 25/150 26/26 [==============================] - 0s 3ms/step - loss: 3.2204 - val_loss: 2.3985 Epoch 26/150 26/26 [==============================] - 0s 3ms/step - loss: 2.9976 - val_loss: 2.2438 Epoch 27/150 26/26 [==============================] - 0s 3ms/step - loss: 2.8315 - val_loss: 2.1098 Epoch 28/150 26/26 [==============================] - 0s 4ms/step - loss: 2.6396 - val_loss: 1.9826 Epoch 29/150 26/26 [==============================] - 0s 3ms/step - loss: 2.4550 - val_loss: 1.8729 Epoch 30/150 26/26 [==============================] - 0s 3ms/step - loss: 2.2768 - val_loss: 1.7561 Epoch 31/150 26/26 [==============================] - 0s 3ms/step - loss: 2.1483 - val_loss: 1.6481 Epoch 32/150 26/26 [==============================] - 0s 3ms/step - loss: 1.9833 - val_loss: 1.5522 Epoch 33/150 26/26 [==============================] - 0s 3ms/step - loss: 1.8917 - val_loss: 1.4667 Epoch 34/150 26/26 [==============================] - 0s 2ms/step - loss: 1.7605 - val_loss: 1.3803 Epoch 35/150 26/26 [==============================] - 0s 3ms/step - loss: 1.6919 - val_loss: 1.2980 Epoch 36/150 26/26 [==============================] - 0s 3ms/step - loss: 1.5612 - val_loss: 1.2294 Epoch 37/150 26/26 [==============================] - 0s 3ms/step - loss: 1.4298 - val_loss: 1.1673 Epoch 38/150 26/26 [==============================] - 0s 3ms/step - loss: 1.3693 - val_loss: 1.1098 Epoch 39/150 26/26 [==============================] - 0s 3ms/step - loss: 1.2730 - val_loss: 1.0596 Epoch 40/150 26/26 [==============================] - 0s 3ms/step - loss: 1.2410 - val_loss: 1.0144 Epoch 41/150 26/26 [==============================] - 0s 3ms/step - loss: 1.1696 - val_loss: 0.9734 Epoch 42/150 26/26 [==============================] - 0s 3ms/step - loss: 1.1018 - val_loss: 0.9336 Epoch 43/150 26/26 [==============================] - 0s 3ms/step - loss: 1.0996 - val_loss: 0.8941 Epoch 44/150 26/26 [==============================] - 0s 3ms/step - loss: 1.0444 - val_loss: 0.8566 Epoch 45/150 26/26 [==============================] - 0s 2ms/step - loss: 1.0093 - val_loss: 0.8250 Epoch 46/150 26/26 [==============================] - 0s 3ms/step - loss: 0.9560 - val_loss: 0.7940 Epoch 47/150 26/26 [==============================] - 0s 3ms/step - loss: 0.9246 - val_loss: 0.7668 Epoch 48/150 26/26 [==============================] - 0s 3ms/step - loss: 0.8871 - val_loss: 0.7422 Epoch 49/150 26/26 [==============================] - 0s 3ms/step - loss: 0.8624 - val_loss: 0.7208 Epoch 50/150 26/26 [==============================] - 0s 3ms/step - loss: 0.8504 - val_loss: 0.6969 Epoch 51/150 26/26 [==============================] - 0s 3ms/step - loss: 0.8216 - val_loss: 0.6774 Epoch 52/150 26/26 [==============================] - 0s 2ms/step - loss: 0.7807 - val_loss: 0.6579 Epoch 53/150 26/26 [==============================] - 0s 3ms/step - loss: 0.7791 - val_loss: 0.6415 Epoch 54/150 26/26 [==============================] - 0s 3ms/step - loss: 0.7530 - val_loss: 0.6252 Epoch 55/150 26/26 [==============================] - 0s 3ms/step - loss: 0.7555 - val_loss: 0.6094 Epoch 56/150 26/26 [==============================] - 0s 3ms/step - loss: 0.7126 - val_loss: 0.5968 Epoch 57/150 26/26 [==============================] - 0s 3ms/step - loss: 0.7056 - val_loss: 0.5835 Epoch 58/150 26/26 [==============================] - 0s 3ms/step - loss: 0.6780 - val_loss: 0.5705 Epoch 59/150 26/26 [==============================] - 0s 3ms/step - loss: 0.6753 - val_loss: 0.5591 Epoch 60/150 26/26 [==============================] - 0s 2ms/step - loss: 0.6652 - val_loss: 0.5492 Epoch 61/150 26/26 [==============================] - 0s 3ms/step - loss: 0.6497 - val_loss: 0.5391 Epoch 62/150 26/26 [==============================] - 0s 2ms/step - loss: 0.6346 - val_loss: 0.5291 Epoch 63/150 26/26 [==============================] - 0s 3ms/step - loss: 0.6307 - val_loss: 0.5191 Epoch 64/150 26/26 [==============================] - 0s 3ms/step - loss: 0.6216 - val_loss: 0.5112 Epoch 65/150 26/26 [==============================] - 0s 3ms/step - loss: 0.6010 - val_loss: 0.5032 Epoch 66/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5938 - val_loss: 0.4953 Epoch 67/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5856 - val_loss: 0.4886 Epoch 68/150 26/26 [==============================] - 0s 2ms/step - loss: 0.5739 - val_loss: 0.4822 Epoch 69/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5742 - val_loss: 0.4765 Epoch 70/150 26/26 [==============================] - 0s 6ms/step - loss: 0.5658 - val_loss: 0.4708 Epoch 71/150 26/26 [==============================] - 0s 6ms/step - loss: 0.5552 - val_loss: 0.4655 Epoch 72/150 26/26 [==============================] - 0s 4ms/step - loss: 0.5543 - val_loss: 0.4601 Epoch 73/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5432 - val_loss: 0.4548 Epoch 74/150 26/26 [==============================] - 0s 4ms/step - loss: 0.5299 - val_loss: 0.4505 Epoch 75/150 26/26 [==============================] - 0s 5ms/step - loss: 0.5485 - val_loss: 0.4460 Epoch 76/150 26/26 [==============================] - 0s 4ms/step - loss: 0.5422 - val_loss: 0.4414 Epoch 77/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5347 - val_loss: 0.4377 Epoch 78/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5225 - val_loss: 0.4341 Epoch 79/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5240 - val_loss: 0.4304 Epoch 80/150 26/26 [==============================] - 0s 2ms/step - loss: 0.5240 - val_loss: 0.4268 Epoch 81/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5207 - val_loss: 0.4253 Epoch 82/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5015 - val_loss: 0.4222 Epoch 83/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5093 - val_loss: 0.4197 Epoch 84/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5076 - val_loss: 0.4181 Epoch 85/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5085 - val_loss: 0.4152 Epoch 86/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5005 - val_loss: 0.4139 Epoch 87/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5057 - val_loss: 0.4117 Epoch 88/150 26/26 [==============================] - 0s 3ms/step - loss: 0.5048 - val_loss: 0.4100 Epoch 89/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4987 - val_loss: 0.4083 Epoch 90/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4922 - val_loss: 0.4073 Epoch 91/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4897 - val_loss: 0.4055 Epoch 92/150 26/26 [==============================] - 0s 2ms/step - loss: 0.5011 - val_loss: 0.4043 Epoch 93/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4818 - val_loss: 0.4026 Epoch 94/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4788 - val_loss: 0.4017 Epoch 95/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4767 - val_loss: 0.3998 Epoch 96/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4850 - val_loss: 0.3990 Epoch 97/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4783 - val_loss: 0.3982 Epoch 98/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4798 - val_loss: 0.3976 Epoch 99/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4829 - val_loss: 0.3972 Epoch 100/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4811 - val_loss: 0.3965 Epoch 101/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4803 - val_loss: 0.3961 Epoch 102/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4675 - val_loss: 0.3961 Epoch 103/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4741 - val_loss: 0.3949 Epoch 104/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4722 - val_loss: 0.3949 Epoch 105/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4776 - val_loss: 0.3939 Epoch 106/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4759 - val_loss: 0.3931 Epoch 107/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4755 - val_loss: 0.3934 Epoch 108/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4655 - val_loss: 0.3928 Epoch 109/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4831 - val_loss: 0.3929 Epoch 110/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4705 - val_loss: 0.3930 Epoch 111/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4694 - val_loss: 0.3931 Epoch 112/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4642 - val_loss: 0.3932 Epoch 113/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4719 - val_loss: 0.3948 Epoch 114/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4730 - val_loss: 0.3943 Epoch 115/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4656 - val_loss: 0.3936 Epoch 116/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4739 - val_loss: 0.3937 Epoch 117/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4690 - val_loss: 0.3947 Epoch 118/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4668 - val_loss: 0.3946 Epoch 119/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4687 - val_loss: 0.3944 Epoch 120/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4632 - val_loss: 0.3936 Epoch 121/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4588 - val_loss: 0.3943 Epoch 122/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4665 - val_loss: 0.3942 Epoch 123/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4674 - val_loss: 0.3949 Epoch 124/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4659 - val_loss: 0.3941 Epoch 125/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4680 - val_loss: 0.3950 Epoch 126/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4610 - val_loss: 0.3942 Epoch 127/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4620 - val_loss: 0.3941 Epoch 128/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4632 - val_loss: 0.3941 Epoch 129/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4569 - val_loss: 0.3945 Epoch 130/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4604 - val_loss: 0.3941 Epoch 131/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4598 - val_loss: 0.3930 Epoch 132/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4666 - val_loss: 0.3939 Epoch 133/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4602 - val_loss: 0.3935 Epoch 134/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4616 - val_loss: 0.3936 Epoch 135/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4657 - val_loss: 0.3936 Epoch 136/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4595 - val_loss: 0.3939 Epoch 137/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4693 - val_loss: 0.3942 Epoch 138/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4690 - val_loss: 0.3939 Epoch 139/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4659 - val_loss: 0.3948 Epoch 140/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4546 - val_loss: 0.3932 Epoch 141/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4593 - val_loss: 0.3934 Epoch 142/150 26/26 [==============================] - 0s 2ms/step - loss: 0.4633 - val_loss: 0.3947 Epoch 143/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4673 - val_loss: 0.3942 Epoch 144/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4593 - val_loss: 0.3945 Epoch 145/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4779 - val_loss: 0.3958 Epoch 146/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4595 - val_loss: 0.3946 Epoch 147/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4690 - val_loss: 0.3940 Epoch 148/150 26/26 [==============================] - 0s 4ms/step - loss: 0.4631 - val_loss: 0.3946 Epoch 149/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4497 - val_loss: 0.3943 Epoch 150/150 26/26 [==============================] - 0s 3ms/step - loss: 0.4513 - val_loss: 0.3947
<tensorflow.python.keras.callbacks.History at 0x24db00d3978>
In above model, the loss seems to be more, so we need to tune the model.
Lets change the network and add 1 more layer and change the loss function to see if we are seeing any imporvement in the results. Below cell has code with Outlier dataset
# Initialize Sequential model
model = tf.keras.models.Sequential()
# Normalize input data
model.add(tf.keras.layers.BatchNormalization(input_shape=(11,)))
# The Hidden Layers : Even with only one dense layer, it is observed that performance enhanced drastically.
model.add(Dense(64, kernel_initializer='normal',activation='relu'))
# model.add(Dense(64, kernel_initializer='normal',activation='relu'))
# model.add(Dense(64, kernel_initializer='normal',activation='relu'))
# The Output Layer :
model.add(Dense(1, kernel_initializer='normal',activation='relu'))
Huber = tf.keras.losses.Huber()
# l = tensorflow.keras.losses.LogCosh()
# h = tensorflow.keras.losses.CategoricalHinge()
# mae = tensorflow.keras.losses.MeanAbsoluteError()
# mse = tensorflow.keras.losses.MeanSquaredError()
optimizers = tf.keras.optimizers.Adam(learning_rate=0.001)
# Compile the model - add mean squared error as loss and stochastic gradient descent as optimizer
model.compile(optimizer=optimizers, loss=Huber)
history = model.fit(X_train_Features, y_train_ActualSignal, epochs=100, validation_split=0.35,
batch_size = 32)
Epoch 1/100 26/26 [==============================] - 1s 10ms/step - loss: 5.0689 - val_loss: 4.7124 Epoch 2/100 26/26 [==============================] - 0s 3ms/step - loss: 4.7838 - val_loss: 4.1261 Epoch 3/100 26/26 [==============================] - 0s 3ms/step - loss: 4.2326 - val_loss: 3.5485 Epoch 4/100 26/26 [==============================] - 0s 3ms/step - loss: 3.2623 - val_loss: 2.8534 Epoch 5/100 26/26 [==============================] - 0s 3ms/step - loss: 1.8733 - val_loss: 2.1005 Epoch 6/100 26/26 [==============================] - 0s 3ms/step - loss: 1.0854 - val_loss: 1.5237 Epoch 7/100 26/26 [==============================] - 0s 3ms/step - loss: 0.8255 - val_loss: 1.2585 Epoch 8/100 26/26 [==============================] - 0s 3ms/step - loss: 0.6898 - val_loss: 1.1263 Epoch 9/100 26/26 [==============================] - 0s 3ms/step - loss: 0.5911 - val_loss: 1.0435 Epoch 10/100 26/26 [==============================] - 0s 3ms/step - loss: 0.5513 - val_loss: 0.9488 Epoch 11/100 26/26 [==============================] - 0s 3ms/step - loss: 0.4601 - val_loss: 0.8679 Epoch 12/100 26/26 [==============================] - 0s 3ms/step - loss: 0.4376 - val_loss: 0.7452 Epoch 13/100 26/26 [==============================] - 0s 3ms/step - loss: 0.4041 - val_loss: 0.6704 Epoch 14/100 26/26 [==============================] - 0s 4ms/step - loss: 0.3656 - val_loss: 0.5627 Epoch 15/100 26/26 [==============================] - 0s 3ms/step - loss: 0.3330 - val_loss: 0.4931 Epoch 16/100 26/26 [==============================] - 0s 3ms/step - loss: 0.3102 - val_loss: 0.4044 Epoch 17/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2857 - val_loss: 0.3803 Epoch 18/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2699 - val_loss: 0.3479 Epoch 19/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2594 - val_loss: 0.2907 Epoch 20/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2462 - val_loss: 0.2633 Epoch 21/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2382 - val_loss: 0.2649 Epoch 22/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2377 - val_loss: 0.2370 Epoch 23/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2310 - val_loss: 0.2354 Epoch 24/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2179 - val_loss: 0.2171 Epoch 25/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2225 - val_loss: 0.2076 Epoch 26/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2253 - val_loss: 0.2121 Epoch 27/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2113 - val_loss: 0.2009 Epoch 28/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2074 - val_loss: 0.1989 Epoch 29/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2041 - val_loss: 0.1986 Epoch 30/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2034 - val_loss: 0.1929 Epoch 31/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2059 - val_loss: 0.1939 Epoch 32/100 26/26 [==============================] - 0s 2ms/step - loss: 0.2121 - val_loss: 0.1903 Epoch 33/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2073 - val_loss: 0.1906 Epoch 34/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2076 - val_loss: 0.1937 Epoch 35/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2013 - val_loss: 0.1864 Epoch 36/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2050 - val_loss: 0.1899 Epoch 37/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1989 - val_loss: 0.1944 Epoch 38/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1969 - val_loss: 0.1977 Epoch 39/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2017 - val_loss: 0.1900 Epoch 40/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1978 - val_loss: 0.1873 Epoch 41/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1966 - val_loss: 0.1873 Epoch 42/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1986 - val_loss: 0.1915 Epoch 43/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1956 - val_loss: 0.1894 Epoch 44/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1965 - val_loss: 0.1870 Epoch 45/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1971 - val_loss: 0.1858 Epoch 46/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1940 - val_loss: 0.1881 Epoch 47/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1992 - val_loss: 0.1894 Epoch 48/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1946 - val_loss: 0.1867 Epoch 49/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1925 - val_loss: 0.1864 Epoch 50/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1926 - val_loss: 0.1874 Epoch 51/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1927 - val_loss: 0.1886 Epoch 52/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1922 - val_loss: 0.1855 Epoch 53/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1883 - val_loss: 0.1922 Epoch 54/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1972 - val_loss: 0.1895 Epoch 55/100 26/26 [==============================] - 0s 2ms/step - loss: 0.1925 - val_loss: 0.1889 Epoch 56/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1946 - val_loss: 0.1872 Epoch 57/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1918 - val_loss: 0.1862 Epoch 58/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1867 - val_loss: 0.1872 Epoch 59/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1861 - val_loss: 0.1876 Epoch 60/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1859 - val_loss: 0.1945 Epoch 61/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1892 - val_loss: 0.1877 Epoch 62/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1875 - val_loss: 0.1873 Epoch 63/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1919 - val_loss: 0.1910 Epoch 64/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1894 - val_loss: 0.1903 Epoch 65/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1872 - val_loss: 0.1896 Epoch 66/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1875 - val_loss: 0.1878 Epoch 67/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1850 - val_loss: 0.1890 Epoch 68/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1827 - val_loss: 0.1942 Epoch 69/100 26/26 [==============================] - 0s 2ms/step - loss: 0.1895 - val_loss: 0.1880 Epoch 70/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1824 - val_loss: 0.1901 Epoch 71/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1799 - val_loss: 0.1896 Epoch 72/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1848 - val_loss: 0.1938 Epoch 73/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1855 - val_loss: 0.1870 Epoch 74/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1803 - val_loss: 0.1888 Epoch 75/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1849 - val_loss: 0.1918 Epoch 76/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1797 - val_loss: 0.1906 Epoch 77/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1830 - val_loss: 0.1908 Epoch 78/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1801 - val_loss: 0.1897 Epoch 79/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1779 - val_loss: 0.1920 Epoch 80/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1795 - val_loss: 0.1927 Epoch 81/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1834 - val_loss: 0.1916 Epoch 82/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1805 - val_loss: 0.1930 Epoch 83/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1839 - val_loss: 0.1923 Epoch 84/100 26/26 [==============================] - 0s 2ms/step - loss: 0.1783 - val_loss: 0.1928 Epoch 85/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1806 - val_loss: 0.1919 Epoch 86/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1793 - val_loss: 0.1937 Epoch 87/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1782 - val_loss: 0.1915 Epoch 88/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1774 - val_loss: 0.1915 Epoch 89/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1765 - val_loss: 0.1980 Epoch 90/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1758 - val_loss: 0.1906 Epoch 91/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1727 - val_loss: 0.1947 Epoch 92/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1794 - val_loss: 0.1956 Epoch 93/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1837 - val_loss: 0.1900 Epoch 94/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1882 - val_loss: 0.1942 Epoch 95/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1805 - val_loss: 0.2033 Epoch 96/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1819 - val_loss: 0.1957 Epoch 97/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1771 - val_loss: 0.1947 Epoch 98/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1871 - val_loss: 0.1908 Epoch 99/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1818 - val_loss: 0.1891 Epoch 100/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1812 - val_loss: 0.1969
def ET_model():
# Initialize Sequential model
model = tf.keras.models.Sequential()
# Normalize input data
model.add(tf.keras.layers.BatchNormalization(input_shape=(11,)))
# The Hidden Layers : Even with only one dense layer, it is observed that performance enhanced drastically.
model.add(Dense(64, kernel_initializer='normal',activation='relu'))
# model.add(Dense(64, kernel_initializer='normal',activation='relu'))
# model.add(Dense(64, kernel_initializer='normal',activation='relu'))
# The Output Layer :
model.add(Dense(1, kernel_initializer='normal',activation='relu'))
#There are several loss function which has been tried, among these Huber performance is good and hence using it.
Huber = tf.keras.losses.Huber()
# l = tensorflow.keras.losses.LogCosh()
# h = tensorflow.keras.losses.CategoricalHinge()
# mae = tensorflow.keras.losses.MeanAbsoluteError()
# mse = tensorflow.keras.losses.MeanSquaredError()
#Among optimizers Adam optimizer was provide good performance.
optimizers = tf.keras.optimizers.Adam(learning_rate=0.001)
# Compile the model - add mean squared error as loss and stochastic gradient descent as optimizer
model.compile(optimizer=optimizers, loss=Huber)
return model
model1 = ET_model()
history = model1.fit(X_train_Features, y_train_ActualSignal, epochs=100, validation_split=0.35,
batch_size = 32)
Epoch 1/100 26/26 [==============================] - 1s 9ms/step - loss: 5.0538 - val_loss: 4.8085 Epoch 2/100 26/26 [==============================] - 0s 5ms/step - loss: 4.7777 - val_loss: 4.4154 Epoch 3/100 26/26 [==============================] - 0s 3ms/step - loss: 4.2641 - val_loss: 3.9726 Epoch 4/100 26/26 [==============================] - 0s 5ms/step - loss: 3.3038 - val_loss: 3.3154 Epoch 5/100 26/26 [==============================] - 0s 4ms/step - loss: 1.9934 - val_loss: 2.3182 Epoch 6/100 26/26 [==============================] - 0s 4ms/step - loss: 1.2401 - val_loss: 1.3362 Epoch 7/100 26/26 [==============================] - 0s 4ms/step - loss: 0.9135 - val_loss: 0.8436 Epoch 8/100 26/26 [==============================] - 0s 4ms/step - loss: 0.7344 - val_loss: 0.6545 Epoch 9/100 26/26 [==============================] - 0s 4ms/step - loss: 0.6301 - val_loss: 0.5675 Epoch 10/100 26/26 [==============================] - 0s 4ms/step - loss: 0.5340 - val_loss: 0.5062 Epoch 11/100 26/26 [==============================] - 0s 4ms/step - loss: 0.4940 - val_loss: 0.4448 Epoch 12/100 26/26 [==============================] - 0s 3ms/step - loss: 0.4193 - val_loss: 0.3997 Epoch 13/100 26/26 [==============================] - 0s 4ms/step - loss: 0.3880 - val_loss: 0.3595 Epoch 14/100 26/26 [==============================] - 0s 4ms/step - loss: 0.3500 - val_loss: 0.3287 Epoch 15/100 26/26 [==============================] - 0s 4ms/step - loss: 0.3174 - val_loss: 0.3247 Epoch 16/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2927 - val_loss: 0.2916 Epoch 17/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2828 - val_loss: 0.2803 Epoch 18/100 26/26 [==============================] - 0s 5ms/step - loss: 0.2563 - val_loss: 0.2686 Epoch 19/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2485 - val_loss: 0.2477 Epoch 20/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2369 - val_loss: 0.2455 Epoch 21/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2205 - val_loss: 0.2353 Epoch 22/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2234 - val_loss: 0.2134 Epoch 23/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2219 - val_loss: 0.2105 Epoch 24/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2145 - val_loss: 0.2033 Epoch 25/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2132 - val_loss: 0.2005 Epoch 26/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2205 - val_loss: 0.2039 Epoch 27/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2107 - val_loss: 0.1976 Epoch 28/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2050 - val_loss: 0.2000 Epoch 29/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2054 - val_loss: 0.1932 Epoch 30/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2026 - val_loss: 0.1932 Epoch 31/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2061 - val_loss: 0.1942 Epoch 32/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2093 - val_loss: 0.1944 Epoch 33/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2051 - val_loss: 0.1915 Epoch 34/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2038 - val_loss: 0.1958 Epoch 35/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1997 - val_loss: 0.1927 Epoch 36/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2029 - val_loss: 0.2057 Epoch 37/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2023 - val_loss: 0.1921 Epoch 38/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2011 - val_loss: 0.1933 Epoch 39/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2040 - val_loss: 0.1893 Epoch 40/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2019 - val_loss: 0.1906 Epoch 41/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1967 - val_loss: 0.1878 Epoch 42/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1974 - val_loss: 0.1915 Epoch 43/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1967 - val_loss: 0.1878 Epoch 44/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1968 - val_loss: 0.1907 Epoch 45/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1975 - val_loss: 0.1879 Epoch 46/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1906 - val_loss: 0.1875 Epoch 47/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1926 - val_loss: 0.1886 Epoch 48/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2002 - val_loss: 0.1926 Epoch 49/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1940 - val_loss: 0.1925 Epoch 50/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1971 - val_loss: 0.1883 Epoch 51/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1976 - val_loss: 0.1850 Epoch 52/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1869 - val_loss: 0.1950 Epoch 53/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1957 - val_loss: 0.1904 Epoch 54/100 26/26 [==============================] - 0s 2ms/step - loss: 0.1946 - val_loss: 0.1883 Epoch 55/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1923 - val_loss: 0.1858 Epoch 56/100 26/26 [==============================] - 0s 2ms/step - loss: 0.1881 - val_loss: 0.1853 Epoch 57/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1878 - val_loss: 0.1881 Epoch 58/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1842 - val_loss: 0.1875 Epoch 59/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1905 - val_loss: 0.1876 Epoch 60/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1857 - val_loss: 0.1906 Epoch 61/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1808 - val_loss: 0.1848 Epoch 62/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1826 - val_loss: 0.1851 Epoch 63/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1851 - val_loss: 0.1887 Epoch 64/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1867 - val_loss: 0.1895 Epoch 65/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1887 - val_loss: 0.1868 Epoch 66/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1855 - val_loss: 0.1906 Epoch 67/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1844 - val_loss: 0.1875 Epoch 68/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1776 - val_loss: 0.1862 Epoch 69/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1826 - val_loss: 0.1882 Epoch 70/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1841 - val_loss: 0.1901 Epoch 71/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1773 - val_loss: 0.1847 Epoch 72/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1850 - val_loss: 0.1885 Epoch 73/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1803 - val_loss: 0.1859 Epoch 74/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1822 - val_loss: 0.1857 Epoch 75/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1837 - val_loss: 0.1837 Epoch 76/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1749 - val_loss: 0.1906 Epoch 77/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1824 - val_loss: 0.1909 Epoch 78/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1781 - val_loss: 0.1920 Epoch 79/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1808 - val_loss: 0.1862 Epoch 80/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1786 - val_loss: 0.1944 Epoch 81/100 26/26 [==============================] - 0s 2ms/step - loss: 0.1745 - val_loss: 0.1898 Epoch 82/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1820 - val_loss: 0.1875 Epoch 83/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1814 - val_loss: 0.1901 Epoch 84/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1748 - val_loss: 0.1911 Epoch 85/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1785 - val_loss: 0.1861 Epoch 86/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1710 - val_loss: 0.1897 Epoch 87/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1792 - val_loss: 0.1862 Epoch 88/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1810 - val_loss: 0.1895 Epoch 89/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1760 - val_loss: 0.1888 Epoch 90/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1797 - val_loss: 0.1871 Epoch 91/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1714 - val_loss: 0.1884 Epoch 92/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1735 - val_loss: 0.1858 Epoch 93/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1734 - val_loss: 0.1888 Epoch 94/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1778 - val_loss: 0.1871 Epoch 95/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1777 - val_loss: 0.1885 Epoch 96/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1755 - val_loss: 0.1893 Epoch 97/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1803 - val_loss: 0.1900 Epoch 98/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1841 - val_loss: 0.1901 Epoch 99/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1784 - val_loss: 0.1870 Epoch 100/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1741 - val_loss: 0.1853
Among optimizers Adam optimizer was provide good performance.
Reference :
https://www.analyticsvidhya.com/blog/2021/05/guide-for-loss-function-in-tensorflow/
# plot the model accuracy and validation accuracy
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.legend(['accuracy','validation accuracy'])
<matplotlib.legend.Legend at 0x24dafd4d2b0>
The above graph gives good insight into data, it shows that it is a good fit network.
A good fit exists between an overfit and underfit model. There is minimal gap between the two final loss values.
There is slight generalization gap observed in the above graph
Below cell has code with Outlier removed dataset
modelOutliersRemoved = ET_model()
history = modelOutliersRemoved.fit(
X_train_OutliersRemovedFeatures
,y_train_OutliersRemovedActualSignal
,epochs=100
,validation_split=0.35
,batch_size = 32)
Epoch 1/100 26/26 [==============================] - 1s 9ms/step - loss: 4.9942 - val_loss: 4.2096 Epoch 2/100 26/26 [==============================] - 0s 4ms/step - loss: 4.6653 - val_loss: 3.4615 Epoch 3/100 26/26 [==============================] - 0s 3ms/step - loss: 4.0741 - val_loss: 2.3214 Epoch 4/100 26/26 [==============================] - 0s 3ms/step - loss: 3.0286 - val_loss: 0.7964 Epoch 5/100 26/26 [==============================] - 0s 3ms/step - loss: 1.6074 - val_loss: 0.7518 Epoch 6/100 26/26 [==============================] - 0s 3ms/step - loss: 0.9393 - val_loss: 0.7920 Epoch 7/100 26/26 [==============================] - 0s 3ms/step - loss: 0.7650 - val_loss: 0.4556 Epoch 8/100 26/26 [==============================] - 0s 4ms/step - loss: 0.6742 - val_loss: 0.3785 Epoch 9/100 26/26 [==============================] - 0s 3ms/step - loss: 0.5969 - val_loss: 0.3484 Epoch 10/100 26/26 [==============================] - 0s 3ms/step - loss: 0.5198 - val_loss: 0.3325 Epoch 11/100 26/26 [==============================] - 0s 3ms/step - loss: 0.4779 - val_loss: 0.3115 Epoch 12/100 26/26 [==============================] - 0s 3ms/step - loss: 0.4364 - val_loss: 0.3010 Epoch 13/100 26/26 [==============================] - 0s 4ms/step - loss: 0.4066 - val_loss: 0.2880 Epoch 14/100 26/26 [==============================] - 0s 3ms/step - loss: 0.3541 - val_loss: 0.2843 Epoch 15/100 26/26 [==============================] - 0s 3ms/step - loss: 0.3304 - val_loss: 0.2655 Epoch 16/100 26/26 [==============================] - 0s 3ms/step - loss: 0.3083 - val_loss: 0.2634 Epoch 17/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2937 - val_loss: 0.2649 Epoch 18/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2736 - val_loss: 0.2400 Epoch 19/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2612 - val_loss: 0.2455 Epoch 20/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2581 - val_loss: 0.2295 Epoch 21/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2468 - val_loss: 0.2198 Epoch 22/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2416 - val_loss: 0.2201 Epoch 23/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2329 - val_loss: 0.2132 Epoch 24/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2302 - val_loss: 0.2135 Epoch 25/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2314 - val_loss: 0.2089 Epoch 26/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2285 - val_loss: 0.2039 Epoch 27/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2278 - val_loss: 0.2057 Epoch 28/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2250 - val_loss: 0.2004 Epoch 29/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2204 - val_loss: 0.1999 Epoch 30/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2228 - val_loss: 0.2063 Epoch 31/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2198 - val_loss: 0.1994 Epoch 32/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2124 - val_loss: 0.1951 Epoch 33/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2106 - val_loss: 0.1968 Epoch 34/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2102 - val_loss: 0.1937 Epoch 35/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2114 - val_loss: 0.1919 Epoch 36/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2095 - val_loss: 0.1888 Epoch 37/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2047 - val_loss: 0.1905 Epoch 38/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2038 - val_loss: 0.1879 Epoch 39/100 26/26 [==============================] - 0s 3ms/step - loss: 0.2004 - val_loss: 0.1882 Epoch 40/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2021 - val_loss: 0.1869 Epoch 41/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1982 - val_loss: 0.1869 Epoch 42/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1980 - val_loss: 0.1902 Epoch 43/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2033 - val_loss: 0.1908 Epoch 44/100 26/26 [==============================] - 0s 4ms/step - loss: 0.2025 - val_loss: 0.1847 Epoch 45/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1974 - val_loss: 0.1841 Epoch 46/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1990 - val_loss: 0.1864 Epoch 47/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1937 - val_loss: 0.1846 Epoch 48/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1937 - val_loss: 0.1893 Epoch 49/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1948 - val_loss: 0.1839 Epoch 50/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1972 - val_loss: 0.1820 Epoch 51/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1886 - val_loss: 0.1842 Epoch 52/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1888 - val_loss: 0.1878 Epoch 53/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1956 - val_loss: 0.1829 Epoch 54/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1900 - val_loss: 0.1864 Epoch 55/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1955 - val_loss: 0.1869 Epoch 56/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1901 - val_loss: 0.1851 Epoch 57/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1927 - val_loss: 0.1872 Epoch 58/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1895 - val_loss: 0.1843 Epoch 59/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1870 - val_loss: 0.1833 Epoch 60/100 26/26 [==============================] - 0s 6ms/step - loss: 0.1873 - val_loss: 0.1848 Epoch 61/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1913 - val_loss: 0.1858 Epoch 62/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1823 - val_loss: 0.1849 Epoch 63/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1882 - val_loss: 0.1867 Epoch 64/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1872 - val_loss: 0.1877 Epoch 65/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1931 - val_loss: 0.1850 Epoch 66/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1813 - val_loss: 0.1822 Epoch 67/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1903 - val_loss: 0.1831 Epoch 68/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1842 - val_loss: 0.1843 Epoch 69/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1834 - val_loss: 0.1858 Epoch 70/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1824 - val_loss: 0.1856 Epoch 71/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1857 - val_loss: 0.1876 Epoch 72/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1891 - val_loss: 0.1856 Epoch 73/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1793 - val_loss: 0.1927 Epoch 74/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1865 - val_loss: 0.1920 Epoch 75/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1780 - val_loss: 0.1838 Epoch 76/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1815 - val_loss: 0.1856 Epoch 77/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1774 - val_loss: 0.1848 Epoch 78/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1835 - val_loss: 0.1852 Epoch 79/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1828 - val_loss: 0.1923 Epoch 80/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1814 - val_loss: 0.1860 Epoch 81/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1795 - val_loss: 0.1869 Epoch 82/100 26/26 [==============================] - 0s 5ms/step - loss: 0.1808 - val_loss: 0.1864 Epoch 83/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1805 - val_loss: 0.1872 Epoch 84/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1796 - val_loss: 0.1877 Epoch 85/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1801 - val_loss: 0.1869 Epoch 86/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1790 - val_loss: 0.1845 Epoch 87/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1738 - val_loss: 0.1874 Epoch 88/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1728 - val_loss: 0.1883 Epoch 89/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1807 - val_loss: 0.1868 Epoch 90/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1772 - val_loss: 0.1917 Epoch 91/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1767 - val_loss: 0.1856 Epoch 92/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1746 - val_loss: 0.1870 Epoch 93/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1743 - val_loss: 0.1828 Epoch 94/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1777 - val_loss: 0.1890 Epoch 95/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1818 - val_loss: 0.1881 Epoch 96/100 26/26 [==============================] - 0s 4ms/step - loss: 0.1779 - val_loss: 0.1875 Epoch 97/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1716 - val_loss: 0.1874 Epoch 98/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1787 - val_loss: 0.1871 Epoch 99/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1749 - val_loss: 0.1871 Epoch 100/100 26/26 [==============================] - 0s 3ms/step - loss: 0.1703 - val_loss: 0.1883
modelOutliersRemoved.save("EAndCRegressor")
INFO:tensorflow:Assets written to: EAndCRegressor\assets
# plot the model accuracy and validation accuracy
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.legend(['loss','val_loss'])
<matplotlib.legend.Legend at 0x24db5a5e278>
The above graph gives good insight into data, it shows that it is a good fit network.
A good fit exists between an overfit and underfit model. There is minimal gap between the two final loss values.
There is slight generalization gap observed in the above graph
calculating the score to evaluate the model
Using earlier picked model to evaluate.
from sklearn.metrics import mean_squared_error
# It can be used to reconstruct the model identically.
reconstructed_model = tf.keras.models.load_model("EAndCRegressor")
y_train_pred = reconstructed_model.predict(X_train_OutliersRemovedFeatures)
y_test_pred = reconstructed_model.predict(X_test_OutliersRemovedFeatures)
print("The R2 score on the Train set is:\t{:0.3f}".format(r2_score(y_train_OutliersRemovedActualSignal, y_train_pred)))
print("The R2 score on the Test set is:\t{:0.3f}".format(r2_score(y_test_OutliersRemovedActualSignal, y_test_pred )))
The R2 score on the Train set is: 0.443 The R2 score on the Test set is: 0.380
Lets check the difference in output of the test dataset so we can conclude on the performance of the trained model
dfOP1 = pd.DataFrame(y_test_OutliersRemovedActualSignal, columns = ['y_test_OutliersRemovedActualSignal'])
dfOP2 = pd.DataFrame(y_test_pred, columns = ['y_test_pred'])
dfOP = dfOP1.merge(dfOP2,left_index=True, right_index=True)
dfOP['diff'] = dfOP['y_test_OutliersRemovedActualSignal'] - dfOP['y_test_pred']
dfOP.sort_values(by='diff', inplace=True, ascending=False)
#Checking all the values of the dataframe
with pd.option_context('display.max_rows', None, 'display.max_columns', None): # more options can be specified also
print(dfOP)
y_test_OutliersRemovedActualSignal y_test_pred diff 110 8 5.729073 2.270927 76 8 5.988072 2.011928 17 7 5.399368 1.600632 142 8 6.491849 1.508151 245 7 5.523509 1.476491 103 7 5.760631 1.239369 209 7 5.804908 1.195092 61 7 5.815044 1.184956 223 7 5.815951 1.184049 18 8 6.884304 1.115696 264 7 5.962145 1.037855 215 7 5.963376 1.036624 291 8 6.977399 1.022601 167 6 4.986062 1.013938 303 6 4.986720 1.013280 126 7 6.033947 0.966053 162 7 6.055186 0.944814 235 7 6.078997 0.921003 199 7 6.078997 0.921003 105 7 6.078997 0.921003 2 6 5.086240 0.913760 273 6 5.087643 0.912357 140 7 6.094343 0.905657 170 6 5.097549 0.902451 46 7 6.100284 0.899716 90 7 6.109637 0.890363 201 6 5.118134 0.881866 54 6 5.147338 0.852662 117 6 5.162206 0.837794 114 6 5.171675 0.828325 239 6 5.241475 0.758525 179 6 5.260069 0.739931 115 7 6.263240 0.736760 63 6 5.267351 0.732649 151 7 6.275689 0.724311 166 6 5.276474 0.723526 26 7 6.280365 0.719635 14 6 5.281938 0.718062 173 6 5.286832 0.713168 189 6 5.286832 0.713168 202 6 5.297847 0.702153 302 7 6.308419 0.691581 27 6 5.314685 0.685315 36 7 6.322194 0.677806 243 7 6.344648 0.655352 55 6 5.346345 0.653655 152 6 5.361393 0.638607 160 7 6.396364 0.603636 283 6 5.404229 0.595771 256 6 5.409431 0.590569 247 6 5.416624 0.583376 172 7 6.418910 0.581090 122 6 5.429602 0.570398 147 6 5.432889 0.567111 265 7 6.434341 0.565659 158 6 5.458155 0.541845 197 6 5.459053 0.540947 123 6 5.460364 0.539636 192 6 5.464156 0.535844 15 7 6.468030 0.531970 194 6 5.473333 0.526667 290 6 5.477630 0.522370 73 6 5.482739 0.517261 304 6 5.483301 0.516699 279 6 5.496977 0.503023 242 6 5.500005 0.499995 125 6 5.501473 0.498527 313 6 5.509701 0.490299 171 5 4.516885 0.483115 24 6 5.517581 0.482419 85 7 6.517829 0.482171 220 6 5.518937 0.481063 124 6 5.525578 0.474422 127 6 5.530915 0.469085 50 5 4.536374 0.463626 312 6 5.541246 0.458754 10 7 6.544585 0.455415 238 6 5.549834 0.450166 285 7 6.549947 0.450053 216 6 5.556232 0.443768 310 6 5.586790 0.413210 83 6 5.586836 0.413164 231 5 4.587257 0.412743 219 6 5.588936 0.411064 183 7 6.598122 0.401878 88 7 6.600156 0.399844 297 7 6.604080 0.395920 251 6 5.646726 0.353274 300 5 4.652462 0.347538 42 7 6.652529 0.347471 121 6 5.655409 0.344591 21 6 5.656130 0.343870 20 5 4.662956 0.337044 25 6 5.664095 0.335905 266 6 5.664941 0.335059 218 6 5.669696 0.330304 9 6 5.707707 0.292293 34 6 5.723586 0.276414 148 6 5.723764 0.276236 259 6 5.730567 0.269433 108 6 5.753483 0.246517 311 6 5.757723 0.242277 244 6 5.759725 0.240275 257 6 5.760796 0.239204 134 6 5.764024 0.235976 49 6 5.764024 0.235976 136 5 4.786978 0.213022 0 6 5.789248 0.210752 113 5 4.789742 0.210258 177 5 4.790184 0.209816 262 5 4.790184 0.209816 146 6 5.793556 0.206444 4 6 5.798077 0.201923 150 6 5.802060 0.197940 280 7 6.805399 0.194601 47 6 5.805992 0.194008 275 5 4.809616 0.190384 253 6 5.813410 0.186590 78 6 5.815214 0.184786 176 6 5.817277 0.182723 159 6 5.818156 0.181844 271 6 5.820154 0.179846 104 6 5.821061 0.178939 31 5 4.822652 0.177348 193 7 6.841774 0.158226 196 5 4.850023 0.149977 99 6 5.871033 0.128967 210 7 6.895298 0.104702 227 7 6.901789 0.098211 128 5 4.910267 0.089733 135 5 4.911142 0.088858 111 5 4.929502 0.070498 270 5 4.945593 0.054407 35 5 4.946137 0.053863 139 6 5.952901 0.047099 309 7 6.954175 0.045825 19 5 4.955269 0.044731 204 7 6.955764 0.044236 174 7 6.980464 0.019536 195 5 4.984324 0.015676 221 6 5.986850 0.013150 289 6 5.989431 0.010569 175 6 5.994975 0.005025 268 6 5.995764 0.004236 154 6 5.998212 0.001788 130 6 5.998212 0.001788 72 6 5.998212 0.001788 191 6 6.007111 -0.007111 314 5 5.010444 -0.010444 33 5 5.010444 -0.010444 206 5 5.014931 -0.014931 184 5 5.015078 -0.015078 97 6 6.015660 -0.015660 7 5 5.020602 -0.020602 87 6 6.030038 -0.030038 116 6 6.032101 -0.032101 119 6 6.041781 -0.041781 101 5 5.044855 -0.044855 84 6 6.045657 -0.045657 237 6 6.045657 -0.045657 306 6 6.051309 -0.051309 307 6 6.057531 -0.057531 45 6 6.058014 -0.058014 292 6 6.064872 -0.064872 3 5 5.066466 -0.066466 98 5 5.082190 -0.082190 301 5 5.091978 -0.091978 222 6 6.092956 -0.092956 315 6 6.093116 -0.093116 188 7 7.093833 -0.093833 65 5 5.095011 -0.095011 211 5 5.096942 -0.096942 228 5 5.097757 -0.097757 163 5 5.099522 -0.099522 233 6 6.110336 -0.110336 293 5 5.112184 -0.112184 56 6 6.112390 -0.112390 70 6 6.117048 -0.117048 260 6 6.118133 -0.118133 96 5 5.120134 -0.120134 178 5 5.122324 -0.122324 316 5 5.123350 -0.123350 118 5 5.124320 -0.124320 95 6 6.124925 -0.124925 120 6 6.128695 -0.128695 91 6 6.135561 -0.135561 22 5 5.149912 -0.149912 28 5 5.151241 -0.151241 208 6 6.159861 -0.159861 203 5 5.160623 -0.160623 86 5 5.167253 -0.167253 298 6 6.173450 -0.173450 164 5 5.176420 -0.176420 74 6 6.180098 -0.180098 43 5 5.183570 -0.183570 169 5 5.193357 -0.193357 39 6 6.194271 -0.194271 80 5 5.196090 -0.196090 145 5 5.206810 -0.206810 112 6 6.207176 -0.207176 29 6 6.209574 -0.209574 207 5 5.227834 -0.227834 94 5 5.229176 -0.229176 217 5 5.229904 -0.229904 100 5 5.238708 -0.238708 32 6 6.241244 -0.241244 149 5 5.246050 -0.246050 41 5 5.258528 -0.258528 137 5 5.266344 -0.266344 5 5 5.269756 -0.269756 249 5 5.271276 -0.271276 226 5 5.271372 -0.271372 157 6 6.271754 -0.271754 278 6 6.277509 -0.277509 1 5 5.280141 -0.280141 106 6 6.290808 -0.290808 156 5 5.292725 -0.292725 141 6 6.298514 -0.298514 287 6 6.298514 -0.298514 138 5 5.298538 -0.298538 277 6 6.306506 -0.306506 51 5 5.309669 -0.309669 281 5 5.316684 -0.316684 48 5 5.323241 -0.323241 305 5 5.323880 -0.323880 131 5 5.327471 -0.327471 267 5 5.329154 -0.329154 132 5 5.329841 -0.329841 317 5 5.331490 -0.331490 205 5 5.335044 -0.335044 255 5 5.337827 -0.337827 69 5 5.338501 -0.338501 286 5 5.344288 -0.344288 16 5 5.346904 -0.346904 59 5 5.359894 -0.359894 168 5 5.364431 -0.364431 214 6 6.366697 -0.366697 82 5 5.367498 -0.367498 225 5 5.376967 -0.376967 229 5 5.394859 -0.394859 109 5 5.395131 -0.395131 30 5 5.399154 -0.399154 62 5 5.419888 -0.419888 190 5 5.420002 -0.420002 200 5 5.420531 -0.420531 75 5 5.420968 -0.420968 296 5 5.426022 -0.426022 180 4 4.430252 -0.430252 53 7 7.430626 -0.430626 232 6 6.438191 -0.438191 67 5 5.439505 -0.439505 318 6 6.446610 -0.446610 13 5 5.453167 -0.453167 107 5 5.456635 -0.456635 213 6 6.462026 -0.462026 230 5 5.476003 -0.476003 143 5 5.480221 -0.480221 295 6 6.497687 -0.497687 272 5 5.499296 -0.499296 182 5 5.506281 -0.506281 165 5 5.518493 -0.518493 66 6 6.518819 -0.518819 129 5 5.527719 -0.527719 77 5 5.536660 -0.536660 276 5 5.553078 -0.553078 299 6 6.570073 -0.570073 234 5 5.573519 -0.573519 6 5 5.580735 -0.580735 198 5 5.583839 -0.583839 236 5 5.585995 -0.585995 92 5 5.602010 -0.602010 282 5 5.604315 -0.604315 181 6 6.612142 -0.612142 161 5 5.628014 -0.628014 57 5 5.628249 -0.628249 81 6 6.637254 -0.637254 294 5 5.652264 -0.652264 79 6 6.666485 -0.666485 269 6 6.681518 -0.681518 23 6 6.681630 -0.681630 64 6 6.728186 -0.728186 58 5 5.734284 -0.734284 153 5 5.753718 -0.753718 252 5 5.762165 -0.762165 284 5 5.763206 -0.763206 144 5 5.769193 -0.769193 44 5 5.779414 -0.779414 250 5 5.781335 -0.781335 308 5 5.787009 -0.787009 263 6 6.793380 -0.793380 102 5 5.814868 -0.814868 40 5 5.821107 -0.821107 274 5 5.825658 -0.825658 37 5 5.836228 -0.836228 68 6 6.863628 -0.863628 224 4 4.875299 -0.875299 12 5 5.879331 -0.879331 52 5 5.882969 -0.882969 258 5 5.895565 -0.895565 60 5 5.917975 -0.917975 261 5 5.919204 -0.919204 246 5 5.933970 -0.933970 8 5 5.981561 -0.981561 240 5 6.098751 -1.098751 187 5 6.123220 -1.123220 155 5 6.202941 -1.202941 93 5 6.231843 -1.231843 212 5 6.237118 -1.237118 241 4 5.238320 -1.238320 89 4 5.260212 -1.260212 319 4 5.387620 -1.387620 288 4 5.406301 -1.406301 71 4 5.461568 -1.461568 38 4 5.486247 -1.486247 254 4 5.495498 -1.495498 185 5 6.520278 -1.520278 248 6 7.541933 -1.541933 133 5 6.585382 -1.585382 186 4 5.834538 -1.834538 11 3 5.271151 -2.271151
dfOP.groupby(pd.cut(dfOP['diff'], np.arange(-3,3,0.2))).count()
| y_test_OutliersRemovedActualSignal | y_test_pred | diff | |
|---|---|---|---|
| diff | |||
| (-3.0, -2.8] | 0 | 0 | 0 |
| (-2.8, -2.6] | 0 | 0 | 0 |
| (-2.6, -2.4] | 0 | 0 | 0 |
| (-2.4, -2.2] | 1 | 1 | 1 |
| (-2.2, -2.0] | 0 | 0 | 0 |
| (-2.0, -1.8] | 1 | 1 | 1 |
| (-1.8, -1.6] | 0 | 0 | 0 |
| (-1.6, -1.4] | 7 | 7 | 7 |
| (-1.4, -1.2] | 6 | 6 | 6 |
| (-1.2, -1.0] | 2 | 2 | 2 |
| (-1.0, -0.8] | 13 | 13 | 13 |
| (-0.8, -0.6] | 20 | 20 | 20 |
| (-0.6, -0.4] | 28 | 28 | 28 |
| (-0.4, -0.2] | 44 | 44 | 44 |
| (-0.2, 2.66e-15] | 51 | 51 | 51 |
| (2.66e-15, 0.2] | 34 | 34 | 34 |
| (0.2, 0.4] | 28 | 28 | 28 |
| (0.4, 0.6] | 37 | 37 | 37 |
| (0.6, 0.8] | 18 | 18 | 18 |
| (0.8, 1.0] | 15 | 15 | 15 |
| (1.0, 1.2] | 9 | 9 | 9 |
| (1.2, 1.4] | 1 | 1 | 1 |
| (1.4, 1.6] | 2 | 2 | 2 |
| (1.6, 1.8] | 1 | 1 | 1 |
| (1.8, 2.0] | 0 | 0 | 0 |
| (2.0, 2.2] | 1 | 1 | 1 |
| (2.2, 2.4] | 1 | 1 | 1 |
| (2.4, 2.6] | 0 | 0 | 0 |
| (2.6, 2.8] | 0 | 0 | 0 |
Observation:
As observed in above tables (in above 2 cells), the difference in predicted value and actual value are not having large difference. There are less number of occurances where the difference observed is large. Large difference of 2 and above are very less and same is true for -2 or less. More occurance are observed where in the difference is in less then 0.5 and greater then -0.5. If we had more instances of value 3,4 and 7,8 in target column then it would be easier to determine accuracy.
Even with only one dense hidden layer, it is observed that performance enhanced drastically and hence adding it. Adding more dense layer is not improving performance and hence commented. Adding Batch normilization in between layer was benefical only between first and second layer, subsequent addition were not so benifical.
There are several loss function which has been tried, among these Huber performance is good and hence using it. the Huber loss is a loss function used in robust regression, that is less sensitive to outliers in data than the squared error loss. Huber loss can be really helpful in such cases, as it curves around the minima which decreases the gradient. And it’s more robust to outliers than MSE. Therefore, it combines good properties from both MSE and MAE.
Among optimizers Adam optimizer was provide good performance.
Reference :
https://www.analyticsvidhya.com/blog/2021/05/guide-for-loss-function-in-tensorflow/
DOMAIN: Autonomous Vehicles
BUSINESS CONTEXT: A Recognising multi-digit numbers in photographs captured at street level is an important component of modern-day map making. A classic example of a corpus of such street-level photographs is Google’s Street View imagery composed of hundreds of millions of geo-located 360-degree panoramic images. The ability to automatically transcribe an address number from a geo-located patch of pixels and associate the transcribed number with a known street address helps pinpoint, with a high degree of accuracy, the location of the building it represents. More broadly, recognising numbers in photographs is a problem of interest to the optical character recognition community. While OCR on constrained domains like document processing is well studied, arbitrary multi-character text recognition in photographs is still highly challenging. This difficulty arises due to the wide variability in the visual appearance of text in the wild on account of a large range of fonts, colours, styles, orientations, and character arrangements. The recognition problem is further complicated by environmental factors such as lighting, shadows, specularity, and occlusions as well as by image acquisition factors such as resolution, motion, and focus blurs. In this project, we will use the dataset with images centred around a single digit (many of the images do contain some distractors at the sides). Although we are taking a sample of the data which is simpler, it is more complex than MNIST because of the distractors.
DATA DESCRIPTION: The SVHN is a real-world image dataset for developing machine learning and object recognition algorithms with the minimal requirement on data formatting but comes from a significantly harder, unsolved, real-world problem (recognising digits and numbers in natural scene images). SVHN is obtained from house numbers in Google Street View images. Where the labels for each of this image are the prominent number in that image i.e. 2,6,7 and 4 respectively.The dataset has been provided in the form of h5py files
PROJECT OBJECTIVE: We will build a digit classifier on the SVHN (Street View Housing Number) dataset. Steps and tasks:
1. Import the data.
2. Data pre-processing and visualisation.
3. Design, train, tune and test a neural network image classifier.
Hint: Use best approach to refine and tune the data or the model. Be highly experimental here to get the best accuracy out of the model.
4. Plot the training loss, validation loss vs number of epochs and training accuracy, validation accuracy vs number of epochs plot and write your observations on the same
import numpy as np
import pandas as pd
import h5py
import matplotlib.pyplot as plt
#To enable plotting graphs in Jupyter notebook
%matplotlib inline
import tensorflow as tf
from tensorflow.keras.layers import BatchNormalization, Dropout
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dense
from tensorflow.keras import optimizers
from keras.callbacks import EarlyStopping
#Open the file as readonly
SVHNRawData=h5py.File("./Part-4-Autonomous_Vehicles_SVHN_single_grey1.h5",'r')
listOfDataKeys = list(SVHNRawData.keys())
listOfDataKeys
['X_test', 'X_train', 'X_val', 'y_test', 'y_train', 'y_val']
SVHN_X_trainData = SVHNRawData['X_train']
SVHN_X_valData = SVHNRawData['X_val']
SVHN_X_testData = SVHNRawData['X_test']
SVHN_Y_trainData = SVHNRawData['y_train']
SVHN_Y_valData = SVHNRawData['y_val']
SVHN_Y_testData = SVHNRawData['y_test']
print(SVHN_X_trainData.shape)
print(SVHN_X_valData.shape)
print(SVHN_X_testData.shape)
print(SVHN_Y_trainData.shape)
print(SVHN_Y_testData.shape)
print(SVHN_Y_valData.shape)
(42000, 32, 32) (60000, 32, 32) (18000, 32, 32) (42000,) (18000,) (60000,)
#Checking data type of the dataset
print(SVHN_X_trainData.dtype)
print(SVHN_X_testData.dtype)
print(SVHN_X_valData.dtype)
print(SVHN_Y_trainData.dtype)
print(SVHN_Y_testData.dtype)
print(SVHN_Y_valData.dtype)
np.unique(SVHN_Y_trainData)
float32 float32 float32 uint8 uint8 uint8
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8)
fig=plt.figure(figsize=(8,8))
columns=5
rows=5
for i in range(1, columns*rows+1):
img=SVHN_X_trainData[i]
fig.add_subplot(rows,columns,i)
plt.imshow(img,cmap='gray')
plt.show()
# show the number in the dataset
plt.imshow(SVHN_X_trainData[1],cmap='gray')
plt.show()
print('Label: ', SVHN_Y_trainData[1])
Label: 6
#Reshape data from 2D to 1D -> 32X32 to 1024
SVHN_X_trainData = np.asarray(SVHN_X_trainData).reshape(42000,1024)
SVHN_X_valData = np.asarray(SVHN_X_valData).reshape(60000,1024)
SVHN_X_testData = np.asarray(SVHN_X_testData).reshape(18000,1024)
print(SVHN_X_trainData.shape)
print(SVHN_X_valData.shape)
print(SVHN_X_testData.shape)
(42000, 1024) (60000, 1024) (18000, 1024)
SVHN_Y_trainData = tf.keras.utils.to_categorical(SVHN_Y_trainData, num_classes=10)
SVHN_Y_testData = tf.keras.utils.to_categorical(SVHN_Y_testData, num_classes=10)
SVHN_Y_valData = tf.keras.utils.to_categorical(SVHN_Y_valData, num_classes=10)
print(SVHN_Y_trainData.shape)
print(SVHN_Y_testData.shape)
print(SVHN_Y_valData.shape)
(42000, 10) (18000, 10) (60000, 10)
model = Sequential()
# Input Layer
#Adding Input layer and activation functions ReLU
model.add(Dense(1024, input_shape = (1024, )))
model.add(Activation('relu'))
#Hidden Layer 1
#Adding 1st Hidden layer
model.add(Dense(512))
model.add(Activation('relu'))
#Hidden Layer 2
#Adding 2nd Hidden layer
model.add(Dense(256))
model.add(Activation('relu'))
#Hidden Layer 3
#Adding 3rd Hidden layer
model.add(Dense(128))
model.add(Activation('relu'))
#Hidden Layer 4
#Adding 4th Hidden layer
model.add(Dense(64))
model.add(Activation('relu'))
#Hidden Layer 4
#Adding 5th Hidden layer
model.add(Dense(32))
model.add(Activation('relu'))
#final layer with softmax
model.add(Dense(10))
model.add(Activation('softmax'))
sgd = optimizers.SGD(lr = 0.0001,momentum=0.9)
model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
model.fit(SVHN_X_trainData, SVHN_Y_trainData,validation_data=(SVHN_X_valData,SVHN_Y_valData), batch_size = 300, epochs = 10, verbose = 1)
D:\anaconda\envs\gl\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:375: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead. "The `lr` argument is deprecated, use `learning_rate` instead.")
Epoch 1/10 140/140 [==============================] - 13s 83ms/step - loss: 8.9691 - accuracy: 0.0985 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 2/10 140/140 [==============================] - 10s 69ms/step - loss: 2.3026 - accuracy: 0.0995 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 3/10 140/140 [==============================] - 11s 81ms/step - loss: 2.3026 - accuracy: 0.1020 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 4/10 140/140 [==============================] - 11s 78ms/step - loss: 2.3026 - accuracy: 0.1020 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 5/10 140/140 [==============================] - 11s 78ms/step - loss: 2.3025 - accuracy: 0.1020 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 6/10 140/140 [==============================] - 10s 70ms/step - loss: 2.3025 - accuracy: 0.1020 - val_loss: 2.3025 - val_accuracy: 0.1000 Epoch 7/10 140/140 [==============================] - 11s 76ms/step - loss: 2.3025 - accuracy: 0.1020 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 8/10 140/140 [==============================] - 11s 77ms/step - loss: 2.3025 - accuracy: 0.1019 - val_loss: 2.3025 - val_accuracy: 0.1000 Epoch 9/10 140/140 [==============================] - 11s 77ms/step - loss: 2.3025 - accuracy: 0.1020 - val_loss: 2.3025 - val_accuracy: 0.1001 Epoch 10/10 140/140 [==============================] - 12s 83ms/step - loss: 2.3025 - accuracy: 0.1020 - val_loss: 2.3025 - val_accuracy: 0.1000
<tensorflow.python.keras.callbacks.History at 0x2b6f6975400>
The accuracy is not improving after few epochs, Other parameters can be changed to see if there is improvement in accuracy. Other neural network has to be checked for given data to improvise the accuracy. Several parameters needs to be changed as well.
adam = optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, )
model.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
model.fit(SVHN_X_trainData, SVHN_Y_trainData,validation_data=(SVHN_X_valData,SVHN_Y_valData), batch_size = 300, epochs = 10, verbose = 1)
Epoch 1/10 140/140 [==============================] - 13s 85ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 2/10 140/140 [==============================] - 11s 78ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 3/10 140/140 [==============================] - 11s 81ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 4/10 140/140 [==============================] - 12s 83ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 5/10 140/140 [==============================] - 11s 81ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 6/10 140/140 [==============================] - 11s 80ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 7/10 140/140 [==============================] - 11s 80ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 8/10 140/140 [==============================] - 11s 80ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 9/10 140/140 [==============================] - 11s 79ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 10/10 140/140 [==============================] - 11s 81ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
<tensorflow.python.keras.callbacks.History at 0x2b6f8a2f7f0>
The same is observed in the above cell, wherein the accuracy is not improving. Kernel initialization can be implemented which could aid in improving accuracy.
he_normal initialization works better for layers with ReLu activation.
##Initialize the Artificial Neural Network Classifier
keras_model_IRA = Sequential()
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model_IRA.add(Dense(512, kernel_initializer='he_normal',input_shape = (1024, )))
#Adding Activation function
keras_model_IRA.add(Activation('relu'))
#Hidden Layer 1
#Adding 1st Hidden layer
keras_model_IRA.add(Dense(256, kernel_initializer='he_normal'))
#Adding Activation function
keras_model_IRA.add(Activation('relu'))
#Hidden Layer 2
#Adding 2nd Hidden layer
keras_model_IRA.add(Dense(128, kernel_initializer='he_normal'))
#Adding Activation function
keras_model_IRA.add(Activation('relu'))
#Hidden Layer 3
#Adding 3rd Hidden layer
keras_model_IRA.add(Dense(64, kernel_initializer='he_normal'))
#Adding Activation function
keras_model_IRA.add(Activation('relu'))
#Hidden Layer 4
#Adding 4th Hidden layer
keras_model_IRA.add(Dense(32, kernel_initializer='he_normal'))
#Adding Activation function
keras_model_IRA.add(Activation('relu'))
# Output Layer
#Adding output layer of 10 nodes, which corresponds to each digits
keras_model_IRA.add(Dense(10))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
keras_model_IRA.add(Activation('softmax'))
keras_model_IRA.summary()
Model: "sequential_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_7 (Dense) (None, 512) 524800 _________________________________________________________________ activation_7 (Activation) (None, 512) 0 _________________________________________________________________ dense_8 (Dense) (None, 256) 131328 _________________________________________________________________ activation_8 (Activation) (None, 256) 0 _________________________________________________________________ dense_9 (Dense) (None, 128) 32896 _________________________________________________________________ activation_9 (Activation) (None, 128) 0 _________________________________________________________________ dense_10 (Dense) (None, 64) 8256 _________________________________________________________________ activation_10 (Activation) (None, 64) 0 _________________________________________________________________ dense_11 (Dense) (None, 32) 2080 _________________________________________________________________ activation_11 (Activation) (None, 32) 0 _________________________________________________________________ dense_12 (Dense) (None, 10) 330 _________________________________________________________________ activation_12 (Activation) (None, 10) 0 ================================================================= Total params: 699,690 Trainable params: 699,690 Non-trainable params: 0 _________________________________________________________________
adam = optimizers.Adam(learning_rate=0.001, beta_1=0.7, beta_2=0.799, epsilon=1e-07 )
keras_model_IRA.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
history = keras_model_IRA.fit(SVHN_X_trainData, SVHN_Y_trainData,validation_data=(SVHN_X_valData,SVHN_Y_valData), batch_size = 300, epochs = 100, verbose = 1)
Epoch 1/100 140/140 [==============================] - 6s 35ms/step - loss: 7.7026 - accuracy: 0.0988 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 2/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3027 - accuracy: 0.1011 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 3/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1010 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 4/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 5/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 6/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 7/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 8/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 9/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 10/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 11/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 12/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 13/100 140/140 [==============================] - 4s 29ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 14/100 140/140 [==============================] - 4s 30ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 15/100 140/140 [==============================] - 4s 31ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 16/100 140/140 [==============================] - 4s 30ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 17/100 140/140 [==============================] - 5s 38ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 18/100 140/140 [==============================] - 5s 37ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 19/100 140/140 [==============================] - 6s 44ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 20/100 140/140 [==============================] - 6s 40ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 21/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 22/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 23/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 24/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 25/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 26/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 27/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 28/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 29/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 30/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1016 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 31/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 32/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 33/100 140/140 [==============================] - 4s 31ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 34/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 35/100 140/140 [==============================] - 6s 40ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 36/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 37/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1007 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 38/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 39/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1012 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 40/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 41/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 42/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 43/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 44/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 45/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 46/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 47/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 48/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 49/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 50/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 51/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 52/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 53/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 54/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 55/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 56/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1005 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 57/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 58/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 59/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 60/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 61/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 62/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 63/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 64/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 65/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 66/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 67/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 68/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1017 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 69/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 70/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 71/100 140/140 [==============================] - 5s 38ms/step - loss: 2.3026 - accuracy: 0.1004 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 72/100 140/140 [==============================] - 5s 37ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 73/100 140/140 [==============================] - 5s 38ms/step - loss: 2.3026 - accuracy: 0.1016 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 74/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 75/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 76/100 140/140 [==============================] - 6s 40ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 77/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 78/100 140/140 [==============================] - 5s 37ms/step - loss: 2.3026 - accuracy: 0.1017 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 79/100 140/140 [==============================] - 5s 37ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 80/100 140/140 [==============================] - 5s 37ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 81/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 82/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1014 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 83/100 140/140 [==============================] - 5s 38ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 84/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 85/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 86/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 87/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1011 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 88/100 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 89/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 90/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 91/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 92/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 93/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1010 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 94/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1011 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 95/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1005 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 96/100 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 97/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 98/100 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1010 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 99/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 100/100 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
In above cell, adam optimizer has been used with learning rate of 0.001. Learning rate of 0.0001 with same beta values are tested and the learning is too slow and we got half the accuracy with same number of epochs. So it is better to have a balanced learning rate and the learning rate affects how quickly our model can converge to a local minima, arrive at best accuracy. It's is required to observe the Val_accuracy and accuracy.
# compiling the classifier
keras_model_IRA.compile(optimizer = 'sgd', loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the Training data
keras_model_IRA.fit(SVHN_X_trainData, SVHN_Y_trainData, validation_data=(SVHN_X_valData,SVHN_Y_valData),batch_size = 300, epochs = 10, verbose = 1)
Epoch 1/10 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 2/10 140/140 [==============================] - 5s 32ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 3/10 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 4/10 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 5/10 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 6/10 140/140 [==============================] - 4s 32ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 7/10 140/140 [==============================] - 5s 33ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 8/10 140/140 [==============================] - 4s 32ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 9/10 140/140 [==============================] - 4s 32ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 10/10 140/140 [==============================] - 4s 32ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
<tensorflow.python.keras.callbacks.History at 0x2b6fb80a2e8>
# compiling the classifier
adagrad = optimizers.Adagrad(lr = 0.001,initial_accumulator_value=0.1)
keras_model_IRA.compile(optimizer = adagrad, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the Training data
keras_model_IRA.fit(SVHN_X_trainData, SVHN_Y_trainData,validation_data=(SVHN_X_valData,SVHN_Y_valData), batch_size = 300, epochs = 20, verbose = 1)
Epoch 1/20 140/140 [==============================] - 6s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 2/20 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 3/20 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 4/20 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 5/20 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 6/20 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 7/20 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 8/20 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 9/20 140/140 [==============================] - 6s 40ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 10/20 140/140 [==============================] - 6s 43ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 11/20 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 12/20 140/140 [==============================] - 4s 31ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 13/20 140/140 [==============================] - 4s 31ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 14/20 140/140 [==============================] - 4s 32ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 15/20 140/140 [==============================] - 5s 39ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 16/20 140/140 [==============================] - 5s 37ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 17/20 140/140 [==============================] - 5s 36ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 18/20 140/140 [==============================] - 5s 35ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 19/20 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000 Epoch 20/20 140/140 [==============================] - 5s 34ms/step - loss: 2.3026 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
<tensorflow.python.keras.callbacks.History at 0x2b6fbdc7668>
In above cell, we are checking with different optimizers, sgd and Adagrad are providing good accuracy score. We can have adam as optimizer. For SGD and Adagrad, other parameters can be changed and checked, but observing score it is better to have adam optimizer.
The plot is for the network with name keras_model_IRA. This model is having 4 hidden layer and relu activation with he_normal kernel initialization.
# list all data in history
print(history.history.keys())
# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
There is no improvement in accuracy and the model needs lot of improvisation, either by adding layer or by tuning parameters !!!
This represents some underfit model as the gap between 2 losses are too much, This model needs improvisation.
def mlp_model():
model = Sequential()
model.add(Dense(1024, input_shape = (1024, )))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dense(512))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dense(128))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dense(64))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))
adagrad = optimizers.Adagrad(lr = 0.001)
model.compile(optimizer = adagrad, loss = 'categorical_crossentropy', metrics = ['accuracy'])
return model
#applying early stopping since after around 80 % of accuracy, there are no improvements observed.
es = EarlyStopping(monitor='val_accuracy', mode='max', verbose=1, patience=5)
# history5 = model.fit(SVHN_X_trainData, SVHN_Y_trainData,validation_data=(SVHN_X_valData,SVHN_Y_valData), validation_split=0.25, epochs = 50, verbose = 1, callbacks=[es])
model = mlp_model()
# history = model.fit(SVHN_X_trainData, SVHN_Y_trainData,validation_data=(SVHN_X_valData,SVHN_Y_valData),batch_size = 200, epochs = 100, verbose = 1)
history = model.fit(SVHN_X_trainData, SVHN_Y_trainData,validation_data=(SVHN_X_valData,SVHN_Y_valData), epochs = 20, verbose = 1, callbacks=[es])
Epoch 1/20 1313/1313 [==============================] - 46s 34ms/step - loss: 1.8937 - accuracy: 0.3636 - val_loss: 1.4849 - val_accuracy: 0.5419 Epoch 2/20 1313/1313 [==============================] - 41s 31ms/step - loss: 1.3936 - accuracy: 0.5713 - val_loss: 1.2198 - val_accuracy: 0.6399 Epoch 3/20 1313/1313 [==============================] - 41s 31ms/step - loss: 1.2111 - accuracy: 0.6338 - val_loss: 1.0633 - val_accuracy: 0.6919 Epoch 4/20 1313/1313 [==============================] - 37s 28ms/step - loss: 1.0995 - accuracy: 0.6720 - val_loss: 0.9643 - val_accuracy: 0.7246 Epoch 5/20 1313/1313 [==============================] - 39s 30ms/step - loss: 1.0234 - accuracy: 0.6930 - val_loss: 0.8931 - val_accuracy: 0.7460 Epoch 6/20 1313/1313 [==============================] - 40s 30ms/step - loss: 0.9670 - accuracy: 0.7101 - val_loss: 0.8290 - val_accuracy: 0.7645 Epoch 7/20 1313/1313 [==============================] - 39s 30ms/step - loss: 0.9111 - accuracy: 0.7284 - val_loss: 0.8237 - val_accuracy: 0.7564 Epoch 8/20 1313/1313 [==============================] - 40s 30ms/step - loss: 0.8755 - accuracy: 0.7413 - val_loss: 0.7635 - val_accuracy: 0.7779 Epoch 9/20 1313/1313 [==============================] - 38s 29ms/step - loss: 0.8372 - accuracy: 0.7506 - val_loss: 0.7231 - val_accuracy: 0.7973 Epoch 10/20 1313/1313 [==============================] - 39s 30ms/step - loss: 0.8091 - accuracy: 0.7604 - val_loss: 0.6957 - val_accuracy: 0.8036 Epoch 11/20 1313/1313 [==============================] - 43s 33ms/step - loss: 0.7851 - accuracy: 0.7648 - val_loss: 0.6916 - val_accuracy: 0.7983 Epoch 12/20 1313/1313 [==============================] - 39s 29ms/step - loss: 0.7609 - accuracy: 0.7720 - val_loss: 0.6529 - val_accuracy: 0.8113 Epoch 13/20 1313/1313 [==============================] - 40s 31ms/step - loss: 0.7355 - accuracy: 0.7830 - val_loss: 0.6319 - val_accuracy: 0.8223 Epoch 14/20 1313/1313 [==============================] - 39s 30ms/step - loss: 0.7193 - accuracy: 0.7863 - val_loss: 0.6253 - val_accuracy: 0.8222 Epoch 15/20 1313/1313 [==============================] - 49s 37ms/step - loss: 0.7020 - accuracy: 0.7928 - val_loss: 0.6058 - val_accuracy: 0.8250 Epoch 16/20 1313/1313 [==============================] - 37s 28ms/step - loss: 0.6840 - accuracy: 0.7962 - val_loss: 0.5746 - val_accuracy: 0.8380 Epoch 17/20 1313/1313 [==============================] - 37s 28ms/step - loss: 0.6727 - accuracy: 0.7996 - val_loss: 0.5721 - val_accuracy: 0.8357 Epoch 18/20 1313/1313 [==============================] - 38s 29ms/step - loss: 0.6524 - accuracy: 0.8064 - val_loss: 0.5945 - val_accuracy: 0.8267 Epoch 19/20 1313/1313 [==============================] - 38s 29ms/step - loss: 0.6407 - accuracy: 0.8096 - val_loss: 0.5640 - val_accuracy: 0.8367 Epoch 20/20 1313/1313 [==============================] - 38s 29ms/step - loss: 0.6245 - accuracy: 0.8161 - val_loss: 0.5305 - val_accuracy: 0.8491
In above model we have used Adagrad optimizer with batch normalization, the network is trained with learning rate of 0.001. For each epoch, the model is taking around 40 seconds. We can use other optimizers and check if this can be improved. Early stopping has been introduced to avoid overfitting and stop the model at appropriate training point.
# list all data in history
print(history.history.keys())
# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
Here validation loss is lower than the training one. In this case, it indicates that the validation dataset may be easier for the model to predict than the training dataset. We can increase epoch and run the model again, but it is always better to add batch normalization and check how we can achieve better convergence.
As observed in above graphs, we are arriving at good accuracy and less loss as number of epochs increases. These can be improved by using other optimizers and adding batch normalization layer.
AdaGrad uses the second moment with no decay to deal with sparse features. RMSProp uses the second moment by with a decay rate to speed up from AdaGrad. Adam uses both first and second moments, and is generally the best choice.
reference : https://towardsdatascience.com/a-visual-explanation-of-gradient-descent-methods-momentum-adagrad-rmsprop-adam-f898b102325c
##Initialize the Artificial Neural Network Classifier
##Batch Normalization, Initialization, relu and Adam
keras_model_BIRA = Sequential()
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model_BIRA.add(Dense(512, kernel_initializer='he_normal',input_shape = (1024, )))
#adding batch normalization layer in between
keras_model_BIRA.add(BatchNormalization())
#Adding Activation function
keras_model_BIRA.add(Activation('relu'))
#Hidden Layer 1
#Adding first Hidden layer
keras_model_BIRA.add(Dense(256, kernel_initializer='he_normal'))
#adding batch normalization layer in between
keras_model_BIRA.add(BatchNormalization())
#Adding Activation function
keras_model_BIRA.add(Activation('relu'))
#Hidden Layer 2
#Adding second Hidden layer
keras_model_BIRA.add(Dense(128, kernel_initializer='he_normal'))
#adding batch normalization layer in between
keras_model_BIRA.add(BatchNormalization())
#Adding Activation function
keras_model_BIRA.add(Activation('relu'))
#Hidden Layer 3
#Adding third Hidden layer
keras_model_BIRA.add(Dense(64, kernel_initializer='he_normal'))
#adding batch normalization layer in between
keras_model_BIRA.add(BatchNormalization())
#Adding Activation function
keras_model_BIRA.add(Activation('relu'))
#Hidden Layer 4
#Adding fourth Hidden layer
keras_model_BIRA.add(Dense(32, kernel_initializer='he_normal'))
#adding batch normalization layer in between
keras_model_BIRA.add(BatchNormalization())
#Adding Activation function
keras_model_BIRA.add(Activation('relu'))
# Output Layer
#Adding output layer which is of 10 nodes (digits)
keras_model_BIRA.add(Dense(10))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
keras_model_BIRA.add(Activation('softmax'))
keras_model_BIRA.summary()
Model: "sequential_3" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_18 (Dense) (None, 512) 524800 _________________________________________________________________ batch_normalization_4 (Batch (None, 512) 2048 _________________________________________________________________ activation_18 (Activation) (None, 512) 0 _________________________________________________________________ dense_19 (Dense) (None, 256) 131328 _________________________________________________________________ batch_normalization_5 (Batch (None, 256) 1024 _________________________________________________________________ activation_19 (Activation) (None, 256) 0 _________________________________________________________________ dense_20 (Dense) (None, 128) 32896 _________________________________________________________________ batch_normalization_6 (Batch (None, 128) 512 _________________________________________________________________ activation_20 (Activation) (None, 128) 0 _________________________________________________________________ dense_21 (Dense) (None, 64) 8256 _________________________________________________________________ batch_normalization_7 (Batch (None, 64) 256 _________________________________________________________________ activation_21 (Activation) (None, 64) 0 _________________________________________________________________ dense_22 (Dense) (None, 32) 2080 _________________________________________________________________ batch_normalization_8 (Batch (None, 32) 128 _________________________________________________________________ activation_22 (Activation) (None, 32) 0 _________________________________________________________________ dense_23 (Dense) (None, 10) 330 _________________________________________________________________ activation_23 (Activation) (None, 10) 0 ================================================================= Total params: 703,658 Trainable params: 701,674 Non-trainable params: 1,984 _________________________________________________________________
# es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=2)
es = EarlyStopping(monitor='val_accuracy', mode='max', verbose=1, patience=5)
adam = optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-07 )
# compiling the classifier
keras_model_BIRA.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting Training data
history = keras_model_BIRA.fit(SVHN_X_trainData, SVHN_Y_trainData, validation_data=(SVHN_X_valData,SVHN_Y_valData),batch_size = 300, epochs = 25, verbose = 1, callbacks=[es])
Epoch 1/25 140/140 [==============================] - 8s 41ms/step - loss: 2.1752 - accuracy: 0.2174 - val_loss: 2.1112 - val_accuracy: 0.2486 Epoch 2/25 140/140 [==============================] - 5s 39ms/step - loss: 1.6567 - accuracy: 0.4752 - val_loss: 1.6823 - val_accuracy: 0.4610 Epoch 3/25 140/140 [==============================] - 6s 41ms/step - loss: 1.3642 - accuracy: 0.6082 - val_loss: 1.4298 - val_accuracy: 0.5653 Epoch 4/25 140/140 [==============================] - 6s 40ms/step - loss: 1.1782 - accuracy: 0.6727 - val_loss: 1.2374 - val_accuracy: 0.6489 Epoch 5/25 140/140 [==============================] - 6s 40ms/step - loss: 1.0394 - accuracy: 0.7176 - val_loss: 1.1638 - val_accuracy: 0.6617 Epoch 6/25 140/140 [==============================] - 6s 40ms/step - loss: 0.9333 - accuracy: 0.7458 - val_loss: 1.0232 - val_accuracy: 0.7063 Epoch 7/25 140/140 [==============================] - 6s 46ms/step - loss: 0.8475 - accuracy: 0.7663 - val_loss: 0.9716 - val_accuracy: 0.7288 Epoch 8/25 140/140 [==============================] - 6s 45ms/step - loss: 0.7773 - accuracy: 0.7853 - val_loss: 0.9108 - val_accuracy: 0.7430 Epoch 9/25 140/140 [==============================] - 5s 37ms/step - loss: 0.7146 - accuracy: 0.8002 - val_loss: 0.8901 - val_accuracy: 0.7324 Epoch 10/25 140/140 [==============================] - 5s 36ms/step - loss: 0.6634 - accuracy: 0.8146 - val_loss: 0.9185 - val_accuracy: 0.7244 Epoch 11/25 140/140 [==============================] - 5s 37ms/step - loss: 0.6227 - accuracy: 0.8249 - val_loss: 0.7736 - val_accuracy: 0.7749 Epoch 12/25 140/140 [==============================] - 5s 36ms/step - loss: 0.5786 - accuracy: 0.8379 - val_loss: 0.8869 - val_accuracy: 0.7284 Epoch 13/25 140/140 [==============================] - 5s 37ms/step - loss: 0.5442 - accuracy: 0.8480 - val_loss: 0.8058 - val_accuracy: 0.7515 Epoch 14/25 140/140 [==============================] - 5s 36ms/step - loss: 0.5102 - accuracy: 0.8563 - val_loss: 0.8205 - val_accuracy: 0.7451 Epoch 15/25 140/140 [==============================] - 5s 36ms/step - loss: 0.4834 - accuracy: 0.8646 - val_loss: 0.6865 - val_accuracy: 0.7941 Epoch 16/25 140/140 [==============================] - 6s 40ms/step - loss: 0.4542 - accuracy: 0.8733 - val_loss: 0.7269 - val_accuracy: 0.7775 Epoch 17/25 140/140 [==============================] - 6s 41ms/step - loss: 0.4269 - accuracy: 0.8809 - val_loss: 0.6931 - val_accuracy: 0.7880 Epoch 18/25 140/140 [==============================] - 6s 41ms/step - loss: 0.4043 - accuracy: 0.8863 - val_loss: 0.6830 - val_accuracy: 0.7886 Epoch 19/25 140/140 [==============================] - 6s 41ms/step - loss: 0.3841 - accuracy: 0.8922 - val_loss: 0.7573 - val_accuracy: 0.7644 Epoch 20/25 140/140 [==============================] - 6s 41ms/step - loss: 0.3608 - accuracy: 0.8994 - val_loss: 0.6651 - val_accuracy: 0.7960 Epoch 21/25 140/140 [==============================] - 6s 39ms/step - loss: 0.3463 - accuracy: 0.9035 - val_loss: 0.7317 - val_accuracy: 0.7714 Epoch 22/25 140/140 [==============================] - 5s 39ms/step - loss: 0.3260 - accuracy: 0.9097 - val_loss: 0.8466 - val_accuracy: 0.7356 Epoch 23/25 140/140 [==============================] - 5s 39ms/step - loss: 0.3106 - accuracy: 0.9140 - val_loss: 0.6689 - val_accuracy: 0.7967 Epoch 24/25 140/140 [==============================] - 6s 39ms/step - loss: 0.2912 - accuracy: 0.9203 - val_loss: 0.5677 - val_accuracy: 0.8251 Epoch 25/25 140/140 [==============================] - 5s 39ms/step - loss: 0.2778 - accuracy: 0.9236 - val_loss: 0.7378 - val_accuracy: 0.7713
By Observing the accuracy and Val_accuracy, there be
# list all data in history
print(history.history.keys())
# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
The generalisation gap between the losses are widening. It looks like a case of overfitting. So as to avoid this let's try drop out layer and see how the model converges.
This is to check if we are getting any increase in accuracy.
##Initialize the Artificial Neural Network Classifier
keras_model_BIRAD = Sequential()
dropout_val = 0.25
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model_BIRAD.add(Dense(512, kernel_initializer='he_normal',input_shape = (1024, )))
#Adding BatchNormalization Layer
keras_model_BIRAD.add(BatchNormalization())
#Adding Activation function
keras_model_BIRAD.add(Activation('relu'))
#Adding Dropout Layer
keras_model_BIRAD.add(Dropout(dropout_val))
#Hidden Layer 1
#Adding first Hidden layer
keras_model_BIRAD.add(Dense(256,kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_BIRAD.add(BatchNormalization())
#Adding Activation function
keras_model_BIRAD.add(Activation('relu'))
#Adding Dropout Layer
keras_model_BIRAD.add(Dropout(dropout_val))
#Hidden Layer 2
#Adding second Hidden layer
keras_model_BIRAD.add(Dense(128, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_BIRAD.add(BatchNormalization())
#Adding Activation function
keras_model_BIRAD.add(Activation('relu'))
#Adding Dropout Layer
keras_model_BIRAD.add(Dropout(dropout_val))
#Hidden Layer 3
#Adding third Hidden layer
keras_model_BIRAD.add(Dense(64, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_BIRAD.add(BatchNormalization())
#Adding Activation function
keras_model_BIRAD.add(Activation('relu'))
#Adding Dropout Layer
keras_model_BIRAD.add(Dropout(dropout_val))
#Hidden Layer 4
#Adding fourth Hidden layer
keras_model_BIRAD.add(Dense(32, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_BIRAD.add(BatchNormalization())
#Adding Activation function
keras_model_BIRAD.add(Activation('relu'))
#Adding Dropout Layer
keras_model_BIRAD.add(Dropout(dropout_val))
# Output Layer
#Adding output layer which is of 10 nodes (digidts)
keras_model_BIRAD.add(Dense(10))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
keras_model_BIRAD.add(Activation('softmax'))
keras_model_BIRAD.summary()
Model: "sequential_4" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= dense_24 (Dense) (None, 512) 524800 _________________________________________________________________ batch_normalization_9 (Batch (None, 512) 2048 _________________________________________________________________ activation_24 (Activation) (None, 512) 0 _________________________________________________________________ dropout (Dropout) (None, 512) 0 _________________________________________________________________ dense_25 (Dense) (None, 256) 131328 _________________________________________________________________ batch_normalization_10 (Batc (None, 256) 1024 _________________________________________________________________ activation_25 (Activation) (None, 256) 0 _________________________________________________________________ dropout_1 (Dropout) (None, 256) 0 _________________________________________________________________ dense_26 (Dense) (None, 128) 32896 _________________________________________________________________ batch_normalization_11 (Batc (None, 128) 512 _________________________________________________________________ activation_26 (Activation) (None, 128) 0 _________________________________________________________________ dropout_2 (Dropout) (None, 128) 0 _________________________________________________________________ dense_27 (Dense) (None, 64) 8256 _________________________________________________________________ batch_normalization_12 (Batc (None, 64) 256 _________________________________________________________________ activation_27 (Activation) (None, 64) 0 _________________________________________________________________ dropout_3 (Dropout) (None, 64) 0 _________________________________________________________________ dense_28 (Dense) (None, 32) 2080 _________________________________________________________________ batch_normalization_13 (Batc (None, 32) 128 _________________________________________________________________ activation_28 (Activation) (None, 32) 0 _________________________________________________________________ dropout_4 (Dropout) (None, 32) 0 _________________________________________________________________ dense_29 (Dense) (None, 10) 330 _________________________________________________________________ activation_29 (Activation) (None, 10) 0 ================================================================= Total params: 703,658 Trainable params: 701,674 Non-trainable params: 1,984 _________________________________________________________________
# adam = optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-07 )
# # compiling the classifier
# keras_model_BIRA.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# # Fitting Training data
# history = keras_model_BIRA.fit(SVHN_X_trainData, SVHN_Y_trainData, validation_data=(SVHN_X_valData,SVHN_Y_valData),batch_size = 300, epochs = 25, verbose = 1, callbacks=[es])
# es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=2)
es = EarlyStopping(monitor='val_accuracy', mode='max', verbose=1, patience=5)
#Setting the learning rate in adam
adam = optimizers.Adam(learning_rate = 0.001)
# compiling the ANN classifier
keras_model_BIRAD.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the ANN to the Training data
history_BIRAD = keras_model_BIRAD.fit(
SVHN_X_trainData
, SVHN_Y_trainData
, validation_data=(SVHN_X_valData,SVHN_Y_valData)
, batch_size = 320
, epochs = 100
, verbose = 1
# , callbacks=[es]
)
history = history_BIRAD
Epoch 1/100 132/132 [==============================] - 9s 50ms/step - loss: 2.3419 - accuracy: 0.1546 - val_loss: 2.0902 - val_accuracy: 0.2253 Epoch 2/100 132/132 [==============================] - 6s 46ms/step - loss: 1.8400 - accuracy: 0.3402 - val_loss: 1.7038 - val_accuracy: 0.4040 Epoch 3/100 132/132 [==============================] - 6s 47ms/step - loss: 1.5052 - accuracy: 0.4773 - val_loss: 1.7268 - val_accuracy: 0.4083 Epoch 4/100 132/132 [==============================] - 6s 46ms/step - loss: 1.3240 - accuracy: 0.5524 - val_loss: 1.8936 - val_accuracy: 0.4047 Epoch 5/100 132/132 [==============================] - 6s 47ms/step - loss: 1.2062 - accuracy: 0.6021 - val_loss: 1.3370 - val_accuracy: 0.5502 Epoch 6/100 132/132 [==============================] - 6s 46ms/step - loss: 1.1053 - accuracy: 0.6404 - val_loss: 1.4082 - val_accuracy: 0.5495 Epoch 7/100 132/132 [==============================] - 6s 48ms/step - loss: 1.0310 - accuracy: 0.6733 - val_loss: 1.1462 - val_accuracy: 0.6297 Epoch 8/100 132/132 [==============================] - 6s 46ms/step - loss: 0.9680 - accuracy: 0.6992 - val_loss: 1.0175 - val_accuracy: 0.6738 Epoch 9/100 132/132 [==============================] - 6s 46ms/step - loss: 0.9168 - accuracy: 0.7204 - val_loss: 1.2693 - val_accuracy: 0.5792 Epoch 10/100 132/132 [==============================] - 6s 46ms/step - loss: 0.8738 - accuracy: 0.7360 - val_loss: 0.8815 - val_accuracy: 0.7322 Epoch 11/100 132/132 [==============================] - 6s 46ms/step - loss: 0.8314 - accuracy: 0.7482 - val_loss: 0.8658 - val_accuracy: 0.7371 Epoch 12/100 132/132 [==============================] - 6s 47ms/step - loss: 0.8066 - accuracy: 0.7594 - val_loss: 1.1057 - val_accuracy: 0.6466 Epoch 13/100 132/132 [==============================] - 6s 47ms/step - loss: 0.7847 - accuracy: 0.7639 - val_loss: 0.7689 - val_accuracy: 0.7681 Epoch 14/100 132/132 [==============================] - 6s 47ms/step - loss: 0.7581 - accuracy: 0.7718 - val_loss: 0.9387 - val_accuracy: 0.6872 Epoch 15/100 132/132 [==============================] - 6s 47ms/step - loss: 0.7362 - accuracy: 0.7813 - val_loss: 0.8342 - val_accuracy: 0.7309 Epoch 16/100 132/132 [==============================] - 6s 46ms/step - loss: 0.7114 - accuracy: 0.7883 - val_loss: 1.1329 - val_accuracy: 0.6508 Epoch 17/100 132/132 [==============================] - 6s 46ms/step - loss: 0.6940 - accuracy: 0.7944 - val_loss: 0.9792 - val_accuracy: 0.6891 Epoch 18/100 132/132 [==============================] - 6s 47ms/step - loss: 0.6866 - accuracy: 0.7971 - val_loss: 0.8289 - val_accuracy: 0.7386 Epoch 19/100 132/132 [==============================] - 6s 46ms/step - loss: 0.6612 - accuracy: 0.8045 - val_loss: 0.9699 - val_accuracy: 0.6844 Epoch 20/100 132/132 [==============================] - 6s 47ms/step - loss: 0.6544 - accuracy: 0.8064 - val_loss: 0.8293 - val_accuracy: 0.7360 Epoch 21/100 132/132 [==============================] - 6s 48ms/step - loss: 0.6378 - accuracy: 0.8106 - val_loss: 0.9192 - val_accuracy: 0.6992 Epoch 22/100 132/132 [==============================] - 6s 47ms/step - loss: 0.6350 - accuracy: 0.8140 - val_loss: 0.8409 - val_accuracy: 0.7412 Epoch 23/100 132/132 [==============================] - 6s 47ms/step - loss: 0.6234 - accuracy: 0.8164 - val_loss: 0.8603 - val_accuracy: 0.7260 Epoch 24/100 132/132 [==============================] - 7s 50ms/step - loss: 0.6068 - accuracy: 0.8214 - val_loss: 0.7839 - val_accuracy: 0.7534 Epoch 25/100 132/132 [==============================] - 6s 47ms/step - loss: 0.6028 - accuracy: 0.8228 - val_loss: 0.7040 - val_accuracy: 0.7823 Epoch 26/100 132/132 [==============================] - 6s 46ms/step - loss: 0.5979 - accuracy: 0.8245 - val_loss: 0.8617 - val_accuracy: 0.7252 Epoch 27/100 132/132 [==============================] - 6s 47ms/step - loss: 0.5795 - accuracy: 0.8277 - val_loss: 0.8938 - val_accuracy: 0.7164 Epoch 28/100 132/132 [==============================] - 6s 47ms/step - loss: 0.5679 - accuracy: 0.8334 - val_loss: 0.9970 - val_accuracy: 0.6967 Epoch 29/100 132/132 [==============================] - 6s 49ms/step - loss: 0.5644 - accuracy: 0.8339 - val_loss: 0.8695 - val_accuracy: 0.7148 Epoch 30/100 132/132 [==============================] - 7s 53ms/step - loss: 0.5638 - accuracy: 0.8362 - val_loss: 0.8300 - val_accuracy: 0.7398 Epoch 31/100 132/132 [==============================] - 7s 51ms/step - loss: 0.5488 - accuracy: 0.8394 - val_loss: 0.8481 - val_accuracy: 0.7260 Epoch 32/100 132/132 [==============================] - 8s 58ms/step - loss: 0.5502 - accuracy: 0.8401 - val_loss: 0.8918 - val_accuracy: 0.7193 Epoch 33/100 132/132 [==============================] - 7s 55ms/step - loss: 0.5432 - accuracy: 0.8415 - val_loss: 0.6436 - val_accuracy: 0.8016 Epoch 34/100 132/132 [==============================] - 6s 46ms/step - loss: 0.5312 - accuracy: 0.8453 - val_loss: 0.7158 - val_accuracy: 0.7883 Epoch 35/100 132/132 [==============================] - 6s 45ms/step - loss: 0.5251 - accuracy: 0.8477 - val_loss: 0.7622 - val_accuracy: 0.7526 Epoch 36/100 132/132 [==============================] - 6s 48ms/step - loss: 0.5156 - accuracy: 0.8491 - val_loss: 0.6965 - val_accuracy: 0.7887 Epoch 37/100 132/132 [==============================] - 6s 48ms/step - loss: 0.5187 - accuracy: 0.8473 - val_loss: 0.7402 - val_accuracy: 0.7659 Epoch 38/100 132/132 [==============================] - 6s 48ms/step - loss: 0.5134 - accuracy: 0.8505 - val_loss: 0.6431 - val_accuracy: 0.7995 Epoch 39/100 132/132 [==============================] - 6s 48ms/step - loss: 0.5071 - accuracy: 0.8516 - val_loss: 0.6769 - val_accuracy: 0.7859 Epoch 40/100 132/132 [==============================] - 6s 48ms/step - loss: 0.5087 - accuracy: 0.8523 - val_loss: 0.6532 - val_accuracy: 0.7960 Epoch 41/100 132/132 [==============================] - 6s 47ms/step - loss: 0.5067 - accuracy: 0.8518 - val_loss: 0.8594 - val_accuracy: 0.7315 Epoch 42/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4910 - accuracy: 0.8576 - val_loss: 0.5907 - val_accuracy: 0.8139 Epoch 43/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4853 - accuracy: 0.8584 - val_loss: 0.7408 - val_accuracy: 0.7633 Epoch 44/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4873 - accuracy: 0.8578 - val_loss: 0.8591 - val_accuracy: 0.7204 Epoch 45/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4732 - accuracy: 0.8594 - val_loss: 0.7186 - val_accuracy: 0.7768 Epoch 46/100 132/132 [==============================] - 6s 48ms/step - loss: 0.4682 - accuracy: 0.8622 - val_loss: 0.8602 - val_accuracy: 0.7196 Epoch 47/100 132/132 [==============================] - 6s 48ms/step - loss: 0.4656 - accuracy: 0.8617 - val_loss: 0.6529 - val_accuracy: 0.8000 Epoch 48/100 132/132 [==============================] - 6s 48ms/step - loss: 0.4645 - accuracy: 0.8650 - val_loss: 0.7317 - val_accuracy: 0.7649 Epoch 49/100 132/132 [==============================] - 6s 48ms/step - loss: 0.4621 - accuracy: 0.8659 - val_loss: 0.8116 - val_accuracy: 0.7267 Epoch 50/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4565 - accuracy: 0.8677 - val_loss: 0.6806 - val_accuracy: 0.7851 Epoch 51/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4563 - accuracy: 0.8679 - val_loss: 0.7933 - val_accuracy: 0.7476 Epoch 52/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4491 - accuracy: 0.8686 - val_loss: 0.7425 - val_accuracy: 0.7571 Epoch 53/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4428 - accuracy: 0.8701 - val_loss: 0.7168 - val_accuracy: 0.7799 Epoch 54/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4397 - accuracy: 0.8711 - val_loss: 0.6678 - val_accuracy: 0.7896 Epoch 55/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4402 - accuracy: 0.8700 - val_loss: 0.9042 - val_accuracy: 0.7105 Epoch 56/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4348 - accuracy: 0.8722 - val_loss: 0.8123 - val_accuracy: 0.7344 Epoch 57/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4376 - accuracy: 0.8716 - val_loss: 0.7193 - val_accuracy: 0.7746 Epoch 58/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4257 - accuracy: 0.8756 - val_loss: 0.6030 - val_accuracy: 0.8081 Epoch 59/100 132/132 [==============================] - 6s 48ms/step - loss: 0.4292 - accuracy: 0.8736 - val_loss: 0.6743 - val_accuracy: 0.7793 Epoch 60/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4257 - accuracy: 0.8751 - val_loss: 0.7103 - val_accuracy: 0.7681 Epoch 61/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4244 - accuracy: 0.8754 - val_loss: 0.7858 - val_accuracy: 0.7451 Epoch 62/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4224 - accuracy: 0.8766 - val_loss: 0.6267 - val_accuracy: 0.8080 Epoch 63/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4214 - accuracy: 0.8775 - val_loss: 0.5737 - val_accuracy: 0.8191 Epoch 64/100 132/132 [==============================] - 6s 46ms/step - loss: 0.4148 - accuracy: 0.8800 - val_loss: 0.6299 - val_accuracy: 0.8118 Epoch 65/100 132/132 [==============================] - 7s 55ms/step - loss: 0.4050 - accuracy: 0.8823 - val_loss: 0.6970 - val_accuracy: 0.7858 Epoch 66/100 132/132 [==============================] - 6s 49ms/step - loss: 0.4137 - accuracy: 0.8782 - val_loss: 0.5429 - val_accuracy: 0.8342 Epoch 67/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4086 - accuracy: 0.8792 - val_loss: 0.6051 - val_accuracy: 0.8075 Epoch 68/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4063 - accuracy: 0.8812 - val_loss: 0.6787 - val_accuracy: 0.7832 Epoch 69/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4064 - accuracy: 0.8813 - val_loss: 0.6095 - val_accuracy: 0.8101 Epoch 70/100 132/132 [==============================] - 6s 47ms/step - loss: 0.4050 - accuracy: 0.8813 - val_loss: 0.6083 - val_accuracy: 0.8057 Epoch 71/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3911 - accuracy: 0.8857 - val_loss: 0.5602 - val_accuracy: 0.8256 Epoch 72/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3933 - accuracy: 0.8851 - val_loss: 0.6443 - val_accuracy: 0.7962 Epoch 73/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3902 - accuracy: 0.8860 - val_loss: 0.6505 - val_accuracy: 0.7885 Epoch 74/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3896 - accuracy: 0.8871 - val_loss: 0.5727 - val_accuracy: 0.8177 Epoch 75/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3907 - accuracy: 0.8859 - val_loss: 0.6956 - val_accuracy: 0.7768 Epoch 76/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3848 - accuracy: 0.8888 - val_loss: 0.5595 - val_accuracy: 0.8238 Epoch 77/100 132/132 [==============================] - 6s 49ms/step - loss: 0.3825 - accuracy: 0.8885 - val_loss: 0.5277 - val_accuracy: 0.8404 Epoch 78/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3774 - accuracy: 0.8887 - val_loss: 0.5540 - val_accuracy: 0.8221 Epoch 79/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3750 - accuracy: 0.8913 - val_loss: 0.4658 - val_accuracy: 0.8552 Epoch 80/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3794 - accuracy: 0.8872 - val_loss: 0.6472 - val_accuracy: 0.7916 Epoch 81/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3720 - accuracy: 0.8915 - val_loss: 0.6484 - val_accuracy: 0.7907 Epoch 82/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3718 - accuracy: 0.8924 - val_loss: 0.5234 - val_accuracy: 0.8388 Epoch 83/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3628 - accuracy: 0.8927 - val_loss: 0.8132 - val_accuracy: 0.7333 Epoch 84/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3612 - accuracy: 0.8940 - val_loss: 0.7582 - val_accuracy: 0.7542 Epoch 85/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3595 - accuracy: 0.8941 - val_loss: 0.4922 - val_accuracy: 0.8444 Epoch 86/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3576 - accuracy: 0.8957 - val_loss: 0.4951 - val_accuracy: 0.8471 Epoch 87/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3583 - accuracy: 0.8940 - val_loss: 0.5775 - val_accuracy: 0.8194 Epoch 88/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3605 - accuracy: 0.8952 - val_loss: 0.5059 - val_accuracy: 0.8395 Epoch 89/100 132/132 [==============================] - 7s 50ms/step - loss: 0.3557 - accuracy: 0.8954 - val_loss: 0.5400 - val_accuracy: 0.8294 Epoch 90/100 132/132 [==============================] - 7s 50ms/step - loss: 0.3482 - accuracy: 0.8987 - val_loss: 0.5521 - val_accuracy: 0.8251 Epoch 91/100 132/132 [==============================] - 7s 50ms/step - loss: 0.3561 - accuracy: 0.8966 - val_loss: 0.4872 - val_accuracy: 0.8480 Epoch 92/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3522 - accuracy: 0.8965 - val_loss: 0.6112 - val_accuracy: 0.8063 Epoch 93/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3445 - accuracy: 0.8992 - val_loss: 0.6233 - val_accuracy: 0.7922 Epoch 94/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3459 - accuracy: 0.8991 - val_loss: 0.5760 - val_accuracy: 0.8164 Epoch 95/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3509 - accuracy: 0.8961 - val_loss: 0.5569 - val_accuracy: 0.8242 Epoch 96/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3368 - accuracy: 0.9015 - val_loss: 0.5072 - val_accuracy: 0.8430 Epoch 97/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3380 - accuracy: 0.9010 - val_loss: 0.5093 - val_accuracy: 0.8402 Epoch 98/100 132/132 [==============================] - 6s 48ms/step - loss: 0.3421 - accuracy: 0.9013 - val_loss: 0.6768 - val_accuracy: 0.7771 Epoch 99/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3429 - accuracy: 0.8995 - val_loss: 0.7729 - val_accuracy: 0.7528 Epoch 100/100 132/132 [==============================] - 6s 47ms/step - loss: 0.3308 - accuracy: 0.9022 - val_loss: 0.4977 - val_accuracy: 0.8431
# list all data in history
print(history.history.keys())
# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'Val'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
The loss and accuracy against epoch gives good insight into network behaviour, it provides changes in learning performance over time in terms of experience.. The lower the loss, the better a model. No large deviation observed between train and validation data.
The same model is also tried with adagrad optimizer, but there were no significant improvement observed in the accuracy.
The generalization gap is not so much and model convergence is happenin as expected. we could increase number of epoch and see how it improvises. But for now let's evaluate the model for test data and see check the accuracy.
results_BIRDA = keras_model_BIRAD.evaluate(SVHN_X_testData,SVHN_Y_testData)
print('Val_acc using BatchNorm and Dropout adam : ', results_BIRDA[1])
563/563 [==============================] - 3s 5ms/step - loss: 0.6368 - accuracy: 0.8001 Val_acc using BatchNorm and Dropout adam : 0.8001111149787903
The model is having good accuracy, so let's check the output on test data and compare it with real data
#Showing the image
plt.imshow(SVHN_X_testData[5].reshape(32,32),cmap='gray')
dt = SVHNRawData['y_test']
print(dt[5])
9
#Predicting the digits
keras_model_BIRAD.predict_classes(SVHN_X_testData)[5]
D:\anaconda\envs\gl\lib\site-packages\tensorflow\python\keras\engine\sequential.py:455: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`, if your model does multi-class classification (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype("int32")`, if your model does binary classification (e.g. if it uses a `sigmoid` last-layer activation).
warnings.warn('`model.predict_classes()` is deprecated and '
9
#Showing the image
plt.imshow(SVHN_X_testData[25].reshape(32,32),cmap='gray')
print(dt[25])
2
#Predicting the digits
keras_model_BIRAD.predict_classes(SVHN_X_testData)[25]
2
#Showing the image
plt.imshow(SVHN_X_testData[110].reshape(32,32),cmap='gray')
print(dt[110])
1
#Predicting the digits
keras_model_BIRAD.predict_classes(SVHN_X_testData)[110]
1
Observation:
It is better to use dense network with 3 or 4 hidden layers. In this experiment, it is observed that the accuracy score is good when he_normal kernal_initialization is used with relu. Adding drop out has also helped to improved network accuracy.
Using Batch Normalization has helped to improve the accuracy. It has standardize the inputs to a network, applied to ether the activations of a prior layer. Batch normalization has accelerated training and provided some regularization, there by reducing generalization error.
Initializing random weights through kernal_initialization has improved the network performance a lot. It is also observed that these techiniques has stabilised the network.
We used the predict_classes on some random images which has given right output, this gives confidence on he network which is built and trained.
We tried different optimizers (SGD, adam) on simple network but it was unable to improve accuracy and very struck at one minima and hence accuracy was also stagnant. We improved the network by adding few layers with some weight initialization which has helped to get better accuracy and upon applying different optimizers it help to get better accuracy. By adding batch normalization and dropout, we got better result.